diff --git a/.mailmap b/.mailmap index d8f49483ebd0..7d6fb567ba5e 100644 --- a/.mailmap +++ b/.mailmap @@ -309,7 +309,10 @@ Gokul Sriram Palanisamy Govindaraj Saminathan Guo Ren Guo Ren -Guru Das Srinagesh +Guru Das Srinagesh +Guru Das Srinagesh +Guru Das Srinagesh +Guru Das Srinagesh Gustavo Padovan Gustavo Padovan Hamza Mahfooz @@ -743,6 +746,7 @@ Sathishkumar Muruganandam Satya Priya S.Çağlar Onur Sayali Lokhande +Sean Anderson Sean Christopherson Sean Nyekjaer Sean Tranchetti diff --git a/CREDITS b/CREDITS index a03b00452a1e..dcf7b271782a 100644 --- a/CREDITS +++ b/CREDITS @@ -4570,8 +4570,5 @@ D: MD driver D: EISA/sysfs subsystem S: France -# Don't add your name here, unless you really _are_ after Marc -# alphabetically. Leonard used to be very proud of being the -# last entry, and he'll get positively pissed if he can't even -# be second-to-last. (and this file really _is_ supposed to be -# in alphabetic order) +# Don't add your name here unless you really are last alphabetically. +# (This file is supposed to be kept in alphabetical order by last name.) diff --git a/Documentation/admin-guide/lockup-watchdogs.rst b/Documentation/admin-guide/lockup-watchdogs.rst index 3e09284a8b9b..8f245f4a95b7 100644 --- a/Documentation/admin-guide/lockup-watchdogs.rst +++ b/Documentation/admin-guide/lockup-watchdogs.rst @@ -16,7 +16,7 @@ details), and a compile option, "BOOTPARAM_SOFTLOCKUP_PANIC", are provided for this. A 'hardlockup' is defined as a bug that causes the CPU to loop in -kernel mode for more than 10 seconds (see "Implementation" below for +kernel mode for several seconds (see "Implementation" below for details), without letting other interrupts have a chance to run. Similarly to the softlockup case, the current stack trace is displayed upon detection and the system will stay locked up unless the default @@ -30,39 +30,135 @@ timeout is set through the confusingly named "kernel.panic" sysctl), to cause the system to reboot automatically after a specified amount of time. +Configuration +============= + +A kernel knob is provided that allows administrators to configure +this period. The "watchdog_thresh" parameter (default 10 seconds) +controls the threshold. The right value for a particular environment +is a trade-off between fast response to lockups and detection overhead. + Implementation ============== -The soft and hard lockup detectors are built on top of the hrtimer and -perf subsystems, respectively. A direct consequence of this is that, -in principle, they should work in any architecture where these -subsystems are present. +The soft and hard lockup detectors are built around an hrtimer. +In addition, the softlockup detector regularly schedules a job, and +the hard lockup detector might use Perf/NMI events on architectures +that support it. -A periodic hrtimer runs to generate interrupts and kick the watchdog -job. An NMI perf event is generated every "watchdog_thresh" -(compile-time initialized to 10 and configurable through sysctl of the -same name) seconds to check for hardlockups. If any CPU in the system -does not receive any hrtimer interrupt during that time the -'hardlockup detector' (the handler for the NMI perf event) will -generate a kernel warning or call panic, depending on the -configuration. +Frequency and Heartbeats +------------------------ -The watchdog job runs in a stop scheduling thread that updates a -timestamp every time it is scheduled. If that timestamp is not updated -for 2*watchdog_thresh seconds (the softlockup threshold) the +The core of the detectors is an hrtimer. It serves multiple purposes: + +- schedules watchdog job for the softlockup detector +- bumps the interrupt counter for hardlockup detectors (heartbeat) +- detects softlockups +- detects hardlockups in Buddy mode + +The period of this hrtimer is 2*watchdog_thresh/5, which is 4 seconds +by default. The hrtimer has two or three chances to generate an interrupt +(heartbeat) before the hardlockup detector kicks in. + +Softlockup Detector +------------------- + +The watchdog job is scheduled by the hrtimer and runs in a stop scheduling +thread. It updates a timestamp every time it is scheduled. If that timestamp +is not updated for 2*watchdog_thresh seconds (the softlockup threshold) the 'softlockup detector' (coded inside the hrtimer callback function) will dump useful debug information to the system log, after which it will call panic if it was instructed to do so or resume execution of other kernel code. -The period of the hrtimer is 2*watchdog_thresh/5, which means it has -two or three chances to generate an interrupt before the hardlockup -detector kicks in. +Hardlockup Detector (NMI/Perf) +------------------------------ -As explained above, a kernel knob is provided that allows -administrators to configure the period of the hrtimer and the perf -event. The right value for a particular environment is a trade-off -between fast response to lockups and detection overhead. +On architectures that support NMI (Non-Maskable Interrupt) perf events, +a periodic NMI is generated every "watchdog_thresh" seconds. + +If any CPU in the system does not receive any hrtimer interrupt +(heartbeat) during the "watchdog_thresh" window, the 'hardlockup +detector' (the handler for the NMI perf event) will generate a kernel +warning or call panic. + +**Detection Overhead (NMI):** + +The time to detect a lockup can vary depending on when the lockup +occurs relative to the NMI check window. Examples below assume a watchdog_thresh of 10. + +* **Best Case:** The lockup occurs just before the first heartbeat is + due. The detector will notice the missing hrtimer interrupt almost + immediately during the next check. + + :: + + Time 100.0: cpu 1 heartbeat + Time 100.1: hardlockup_check, cpu1 stores its state + Time 103.9: Hard Lockup on cpu1 + Time 104.0: cpu 1 heartbeat never comes + Time 110.1: hardlockup_check, cpu1 checks the state again, should be the same, declares lockup + + Time to detection: ~6 seconds + +* **Worst Case:** The lockup occurs shortly after a valid interrupt + (heartbeat) which itself happened just after the NMI check. The next + NMI check sees that the interrupt count has changed (due to that one + heartbeat), assumes the CPU is healthy, and resets the baseline. The + lockup is only detected at the subsequent check. + + :: + + Time 100.0: hardlockup_check, cpu1 stores its state + Time 100.1: cpu 1 heartbeat + Time 100.2: Hard Lockup on cpu1 + Time 110.0: hardlockup_check, cpu1 stores its state (misses lockup as state changed) + Time 120.0: hardlockup_check, cpu1 checks the state again, should be the same, declares lockup + + Time to detection: ~20 seconds + +Hardlockup Detector (Buddy) +--------------------------- + +On architectures or configurations where NMI perf events are not +available (or disabled), the kernel may use the "buddy" hardlockup +detector. This mechanism requires SMP (Symmetric Multi-Processing). + +In this mode, each CPU is assigned a "buddy" CPU to monitor. The +monitoring CPU runs its own hrtimer (the same one used for softlockup +detection) and checks if the buddy CPU's hrtimer interrupt count has +increased. + +To ensure timeliness and avoid false positives, the buddy system performs +checks at every hrtimer interval (2*watchdog_thresh/5, which is 4 seconds +by default). It uses a missed-interrupt threshold of 3. If the buddy's +interrupt count has not changed for 3 consecutive checks, it is assumed +that the buddy CPU is hardlocked (interrupts disabled). The monitoring +CPU will then trigger the hardlockup response (warning or panic). + +**Detection Overhead (Buddy):** + +With a default check interval of 4 seconds (watchdog_thresh = 10): + +* **Best case:** Lockup occurs just before a check. + Detected in ~8s (0s till 1st check + 4s till 2nd + 4s till 3rd). +* **Worst case:** Lockup occurs just after a check. + Detected in ~12s (4s till 1st check + 4s till 2nd + 4s till 3rd). + +**Limitations of the Buddy Detector:** + +1. **All-CPU Lockup:** If all CPUs lock up simultaneously, the buddy + detector cannot detect the condition because the monitoring CPUs + are also frozen. +2. **Stack Traces:** Unlike the NMI detector, the buddy detector + cannot directly interrupt the locked CPU to grab a stack trace. + It relies on architecture-specific mechanisms (like NMI backtrace + support) to try and retrieve the status of the locked CPU. If + such support is missing, the log may only show that a lockup + occurred without providing the locked CPU's stack. + +Watchdog Core Exclusion +======================= By default, the watchdog runs on all online cores. However, on a kernel configured with NO_HZ_FULL, by default the watchdog runs only diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index 9aed74e65cf4..c6994e55d141 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -418,7 +418,8 @@ hung_task_detect_count ====================== Indicates the total number of tasks that have been detected as hung since -the system boot. +the system boot or since the counter was reset. The counter is zeroed when +a value of 0 is written. This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. diff --git a/Documentation/devicetree/bindings/timer/xlnx,xps-timer.yaml b/Documentation/devicetree/bindings/timer/xlnx,xps-timer.yaml index b1597db04263..3538eafff6b1 100644 --- a/Documentation/devicetree/bindings/timer/xlnx,xps-timer.yaml +++ b/Documentation/devicetree/bindings/timer/xlnx,xps-timer.yaml @@ -7,7 +7,7 @@ $schema: http://devicetree.org/meta-schemas/core.yaml# title: Xilinx LogiCORE IP AXI Timer maintainers: - - Sean Anderson + - Sean Anderson properties: compatible: diff --git a/MAINTAINERS b/MAINTAINERS index 54b33d43e9fd..387d150a6f7f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -35,6 +35,7 @@ Descriptions of section entries and preferred order F: drivers/net/ all files in and below drivers/net F: drivers/net/* all files in drivers/net, but not below F: */net/* all files in "any top level directory"/net + F: fs/**/*foo*.c all *foo*.c files in any subdirectory of fs One pattern per line. Multiple F: lines acceptable. X: *Excluded* files and directories that are NOT maintained, same rules as F:. Files exclusions are tested before file matches. @@ -10308,7 +10309,7 @@ F: drivers/net/ethernet/freescale/dpaa FREESCALE QORIQ DPAA FMAN DRIVER M: Madalin Bucur -R: Sean Anderson +R: Sean Anderson L: netdev@vger.kernel.org S: Maintained F: Documentation/devicetree/bindings/net/fsl,fman*.yaml @@ -29109,7 +29110,7 @@ S: Orphan F: drivers/net/ethernet/xilinx/ll_temac* XILINX PWM DRIVER -M: Sean Anderson +M: Sean Anderson S: Maintained F: drivers/pwm/pwm-xilinx.c F: include/clocksource/timer-xilinx.h diff --git a/arch/arm/lib/Makefile b/arch/arm/lib/Makefile index 0ca5aae1bcc3..9295055cdfc9 100644 --- a/arch/arm/lib/Makefile +++ b/arch/arm/lib/Makefile @@ -39,9 +39,4 @@ endif $(obj)/csumpartialcopy.o: $(obj)/csumpartialcopygeneric.S $(obj)/csumpartialcopyuser.o: $(obj)/csumpartialcopygeneric.S -ifeq ($(CONFIG_KERNEL_MODE_NEON),y) - CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) - obj-$(CONFIG_XOR_BLOCKS) += xor-neon.o -endif - obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o diff --git a/arch/arm64/include/asm/xor.h b/arch/arm64/include/asm/xor.h deleted file mode 100644 index c38e3d017a79..000000000000 --- a/arch/arm64/include/asm/xor.h +++ /dev/null @@ -1,73 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * arch/arm64/include/asm/xor.h - * - * Authors: Jackie Liu - * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd. - */ - -#include -#include -#include -#include - -#ifdef CONFIG_KERNEL_MODE_NEON - -extern struct xor_block_template const xor_block_inner_neon; - -static void -xor_neon_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2) -{ - scoped_ksimd() - xor_block_inner_neon.do_2(bytes, p1, p2); -} - -static void -xor_neon_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3) -{ - scoped_ksimd() - xor_block_inner_neon.do_3(bytes, p1, p2, p3); -} - -static void -xor_neon_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4) -{ - scoped_ksimd() - xor_block_inner_neon.do_4(bytes, p1, p2, p3, p4); -} - -static void -xor_neon_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5) -{ - scoped_ksimd() - xor_block_inner_neon.do_5(bytes, p1, p2, p3, p4, p5); -} - -static struct xor_block_template xor_block_arm64 = { - .name = "arm64_neon", - .do_2 = xor_neon_2, - .do_3 = xor_neon_3, - .do_4 = xor_neon_4, - .do_5 = xor_neon_5 -}; -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ - do { \ - xor_speed(&xor_block_8regs); \ - xor_speed(&xor_block_32regs); \ - if (cpu_has_neon()) { \ - xor_speed(&xor_block_arm64);\ - } \ - } while (0) - -#endif /* ! CONFIG_KERNEL_MODE_NEON */ diff --git a/arch/arm64/kernel/machine_kexec_file.c b/arch/arm64/kernel/machine_kexec_file.c index fba260ad87a9..e31fabed378a 100644 --- a/arch/arm64/kernel/machine_kexec_file.c +++ b/arch/arm64/kernel/machine_kexec_file.c @@ -134,6 +134,10 @@ int load_other_segments(struct kimage *image, kexec_dprintk("Loaded elf core header at 0x%lx bufsz=0x%lx memsz=0x%lx\n", image->elf_load_addr, kbuf.bufsz, kbuf.memsz); + + ret = crash_load_dm_crypt_keys(image); + if (ret) + goto out_err; } #endif diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 633e5223d944..448c917494f3 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -5,12 +5,6 @@ lib-y := clear_user.o delay.o copy_from_user.o \ memset.o memcmp.o strcmp.o strncmp.o strlen.o \ strnlen.o strchr.o strrchr.o tishift.o -ifeq ($(CONFIG_KERNEL_MODE_NEON), y) -obj-$(CONFIG_XOR_BLOCKS) += xor-neon.o -CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) -CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU) -endif - lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o diff --git a/arch/loongarch/include/asm/xor.h b/arch/loongarch/include/asm/xor.h deleted file mode 100644 index 12467fffee46..000000000000 --- a/arch/loongarch/include/asm/xor.h +++ /dev/null @@ -1,68 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * Copyright (C) 2023 WANG Xuerui - */ -#ifndef _ASM_LOONGARCH_XOR_H -#define _ASM_LOONGARCH_XOR_H - -#include -#include - -#ifdef CONFIG_CPU_HAS_LSX -static struct xor_block_template xor_block_lsx = { - .name = "lsx", - .do_2 = xor_lsx_2, - .do_3 = xor_lsx_3, - .do_4 = xor_lsx_4, - .do_5 = xor_lsx_5, -}; - -#define XOR_SPEED_LSX() \ - do { \ - if (cpu_has_lsx) \ - xor_speed(&xor_block_lsx); \ - } while (0) -#else /* CONFIG_CPU_HAS_LSX */ -#define XOR_SPEED_LSX() -#endif /* CONFIG_CPU_HAS_LSX */ - -#ifdef CONFIG_CPU_HAS_LASX -static struct xor_block_template xor_block_lasx = { - .name = "lasx", - .do_2 = xor_lasx_2, - .do_3 = xor_lasx_3, - .do_4 = xor_lasx_4, - .do_5 = xor_lasx_5, -}; - -#define XOR_SPEED_LASX() \ - do { \ - if (cpu_has_lasx) \ - xor_speed(&xor_block_lasx); \ - } while (0) -#else /* CONFIG_CPU_HAS_LASX */ -#define XOR_SPEED_LASX() -#endif /* CONFIG_CPU_HAS_LASX */ - -/* - * For grins, also test the generic routines. - * - * More importantly: it cannot be ruled out at this point of time, that some - * future (maybe reduced) models could run the vector algorithms slower than - * the scalar ones, maybe for errata or micro-op reasons. It may be - * appropriate to revisit this after one or two more uarch generations. - */ -#include - -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ -do { \ - xor_speed(&xor_block_8regs); \ - xor_speed(&xor_block_8regs_p); \ - xor_speed(&xor_block_32regs); \ - xor_speed(&xor_block_32regs_p); \ - XOR_SPEED_LSX(); \ - XOR_SPEED_LASX(); \ -} while (0) - -#endif /* _ASM_LOONGARCH_XOR_H */ diff --git a/arch/loongarch/include/asm/xor_simd.h b/arch/loongarch/include/asm/xor_simd.h deleted file mode 100644 index 471b96332f38..000000000000 --- a/arch/loongarch/include/asm/xor_simd.h +++ /dev/null @@ -1,34 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * Copyright (C) 2023 WANG Xuerui - */ -#ifndef _ASM_LOONGARCH_XOR_SIMD_H -#define _ASM_LOONGARCH_XOR_SIMD_H - -#ifdef CONFIG_CPU_HAS_LSX -void xor_lsx_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2); -void xor_lsx_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, const unsigned long * __restrict p3); -void xor_lsx_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, const unsigned long * __restrict p3, - const unsigned long * __restrict p4); -void xor_lsx_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, const unsigned long * __restrict p3, - const unsigned long * __restrict p4, const unsigned long * __restrict p5); -#endif /* CONFIG_CPU_HAS_LSX */ - -#ifdef CONFIG_CPU_HAS_LASX -void xor_lasx_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2); -void xor_lasx_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, const unsigned long * __restrict p3); -void xor_lasx_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, const unsigned long * __restrict p3, - const unsigned long * __restrict p4); -void xor_lasx_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, const unsigned long * __restrict p3, - const unsigned long * __restrict p4, const unsigned long * __restrict p5); -#endif /* CONFIG_CPU_HAS_LASX */ - -#endif /* _ASM_LOONGARCH_XOR_SIMD_H */ diff --git a/arch/loongarch/lib/Makefile b/arch/loongarch/lib/Makefile index ccea3bbd4353..827a88529a42 100644 --- a/arch/loongarch/lib/Makefile +++ b/arch/loongarch/lib/Makefile @@ -8,6 +8,4 @@ lib-y += delay.o memset.o memcpy.o memmove.o \ obj-$(CONFIG_ARCH_SUPPORTS_INT128) += tishift.o -obj-$(CONFIG_CPU_HAS_LSX) += xor_simd.o xor_simd_glue.o - obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o diff --git a/arch/loongarch/lib/xor_simd_glue.c b/arch/loongarch/lib/xor_simd_glue.c deleted file mode 100644 index 393f689dbcf6..000000000000 --- a/arch/loongarch/lib/xor_simd_glue.c +++ /dev/null @@ -1,72 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * LoongArch SIMD XOR operations - * - * Copyright (C) 2023 WANG Xuerui - */ - -#include -#include -#include -#include -#include "xor_simd.h" - -#define MAKE_XOR_GLUE_2(flavor) \ -void xor_##flavor##_2(unsigned long bytes, unsigned long * __restrict p1, \ - const unsigned long * __restrict p2) \ -{ \ - kernel_fpu_begin(); \ - __xor_##flavor##_2(bytes, p1, p2); \ - kernel_fpu_end(); \ -} \ -EXPORT_SYMBOL_GPL(xor_##flavor##_2) - -#define MAKE_XOR_GLUE_3(flavor) \ -void xor_##flavor##_3(unsigned long bytes, unsigned long * __restrict p1, \ - const unsigned long * __restrict p2, \ - const unsigned long * __restrict p3) \ -{ \ - kernel_fpu_begin(); \ - __xor_##flavor##_3(bytes, p1, p2, p3); \ - kernel_fpu_end(); \ -} \ -EXPORT_SYMBOL_GPL(xor_##flavor##_3) - -#define MAKE_XOR_GLUE_4(flavor) \ -void xor_##flavor##_4(unsigned long bytes, unsigned long * __restrict p1, \ - const unsigned long * __restrict p2, \ - const unsigned long * __restrict p3, \ - const unsigned long * __restrict p4) \ -{ \ - kernel_fpu_begin(); \ - __xor_##flavor##_4(bytes, p1, p2, p3, p4); \ - kernel_fpu_end(); \ -} \ -EXPORT_SYMBOL_GPL(xor_##flavor##_4) - -#define MAKE_XOR_GLUE_5(flavor) \ -void xor_##flavor##_5(unsigned long bytes, unsigned long * __restrict p1, \ - const unsigned long * __restrict p2, \ - const unsigned long * __restrict p3, \ - const unsigned long * __restrict p4, \ - const unsigned long * __restrict p5) \ -{ \ - kernel_fpu_begin(); \ - __xor_##flavor##_5(bytes, p1, p2, p3, p4, p5); \ - kernel_fpu_end(); \ -} \ -EXPORT_SYMBOL_GPL(xor_##flavor##_5) - -#define MAKE_XOR_GLUES(flavor) \ - MAKE_XOR_GLUE_2(flavor); \ - MAKE_XOR_GLUE_3(flavor); \ - MAKE_XOR_GLUE_4(flavor); \ - MAKE_XOR_GLUE_5(flavor) - -#ifdef CONFIG_CPU_HAS_LSX -MAKE_XOR_GLUES(lsx); -#endif - -#ifdef CONFIG_CPU_HAS_LASX -MAKE_XOR_GLUES(lasx); -#endif diff --git a/arch/powerpc/include/asm/xor.h b/arch/powerpc/include/asm/xor.h deleted file mode 100644 index 37d05c11d09c..000000000000 --- a/arch/powerpc/include/asm/xor.h +++ /dev/null @@ -1,47 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * - * Copyright (C) IBM Corporation, 2012 - * - * Author: Anton Blanchard - */ -#ifndef _ASM_POWERPC_XOR_H -#define _ASM_POWERPC_XOR_H - -#ifdef CONFIG_ALTIVEC - -#include -#include -#include - -static struct xor_block_template xor_block_altivec = { - .name = "altivec", - .do_2 = xor_altivec_2, - .do_3 = xor_altivec_3, - .do_4 = xor_altivec_4, - .do_5 = xor_altivec_5, -}; - -#define XOR_SPEED_ALTIVEC() \ - do { \ - if (cpu_has_feature(CPU_FTR_ALTIVEC)) \ - xor_speed(&xor_block_altivec); \ - } while (0) -#else -#define XOR_SPEED_ALTIVEC() -#endif - -/* Also try the generic routines. */ -#include - -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ -do { \ - xor_speed(&xor_block_8regs); \ - xor_speed(&xor_block_8regs_p); \ - xor_speed(&xor_block_32regs); \ - xor_speed(&xor_block_32regs_p); \ - XOR_SPEED_ALTIVEC(); \ -} while (0) - -#endif /* _ASM_POWERPC_XOR_H */ diff --git a/arch/powerpc/include/asm/xor_altivec.h b/arch/powerpc/include/asm/xor_altivec.h deleted file mode 100644 index 294620a25f80..000000000000 --- a/arch/powerpc/include/asm/xor_altivec.h +++ /dev/null @@ -1,22 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_POWERPC_XOR_ALTIVEC_H -#define _ASM_POWERPC_XOR_ALTIVEC_H - -#ifdef CONFIG_ALTIVEC -void xor_altivec_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2); -void xor_altivec_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3); -void xor_altivec_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4); -void xor_altivec_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5); - -#endif -#endif /* _ASM_POWERPC_XOR_ALTIVEC_H */ diff --git a/arch/powerpc/kexec/elf_64.c b/arch/powerpc/kexec/elf_64.c index 5d6d616404cf..ea50a072debf 100644 --- a/arch/powerpc/kexec/elf_64.c +++ b/arch/powerpc/kexec/elf_64.c @@ -79,6 +79,10 @@ static void *elf64_load(struct kimage *image, char *kernel_buf, goto out; } + ret = crash_load_dm_crypt_keys(image); + if (ret) + goto out; + /* Setup cmdline for kdump kernel case */ modified_cmdline = setup_kdump_cmdline(image, cmdline, cmdline_len); diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile index bcdf387f3998..b1b7a1419653 100644 --- a/arch/powerpc/lib/Makefile +++ b/arch/powerpc/lib/Makefile @@ -72,9 +72,4 @@ obj-$(CONFIG_PPC_LIB_RHEAP) += rheap.o obj-$(CONFIG_FTR_FIXUP_SELFTEST) += feature-fixups-test.o -obj-$(CONFIG_ALTIVEC) += xor_vmx.o xor_vmx_glue.o -CFLAGS_xor_vmx.o += -mhard-float -maltivec $(call cc-option,-mabi=altivec) -# Enable -CFLAGS_xor_vmx.o += -isystem $(shell $(CC) -print-file-name=include) - obj-$(CONFIG_PPC64) += $(obj64-y) diff --git a/arch/powerpc/lib/xor_vmx.h b/arch/powerpc/lib/xor_vmx.h deleted file mode 100644 index 573c41d90dac..000000000000 --- a/arch/powerpc/lib/xor_vmx.h +++ /dev/null @@ -1,22 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * Simple interface to link xor_vmx.c and xor_vmx_glue.c - * - * Separating these file ensures that no altivec instructions are run - * outside of the enable/disable altivec block. - */ - -void __xor_altivec_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2); -void __xor_altivec_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3); -void __xor_altivec_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4); -void __xor_altivec_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5); diff --git a/arch/powerpc/lib/xor_vmx_glue.c b/arch/powerpc/lib/xor_vmx_glue.c deleted file mode 100644 index 35d917ece4d1..000000000000 --- a/arch/powerpc/lib/xor_vmx_glue.c +++ /dev/null @@ -1,63 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * Altivec XOR operations - * - * Copyright 2017 IBM Corp. - */ - -#include -#include -#include -#include -#include -#include "xor_vmx.h" - -void xor_altivec_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2) -{ - preempt_disable(); - enable_kernel_altivec(); - __xor_altivec_2(bytes, p1, p2); - disable_kernel_altivec(); - preempt_enable(); -} -EXPORT_SYMBOL(xor_altivec_2); - -void xor_altivec_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3) -{ - preempt_disable(); - enable_kernel_altivec(); - __xor_altivec_3(bytes, p1, p2, p3); - disable_kernel_altivec(); - preempt_enable(); -} -EXPORT_SYMBOL(xor_altivec_3); - -void xor_altivec_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4) -{ - preempt_disable(); - enable_kernel_altivec(); - __xor_altivec_4(bytes, p1, p2, p3, p4); - disable_kernel_altivec(); - preempt_enable(); -} -EXPORT_SYMBOL(xor_altivec_4); - -void xor_altivec_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5) -{ - preempt_disable(); - enable_kernel_altivec(); - __xor_altivec_5(bytes, p1, p2, p3, p4, p5); - disable_kernel_altivec(); - preempt_enable(); -} -EXPORT_SYMBOL(xor_altivec_5); diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h deleted file mode 100644 index 96011861e46b..000000000000 --- a/arch/riscv/include/asm/xor.h +++ /dev/null @@ -1,68 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * Copyright (C) 2021 SiFive - */ - -#include -#include -#ifdef CONFIG_RISCV_ISA_V -#include -#include -#include - -static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1, - const unsigned long *__restrict p2) -{ - kernel_vector_begin(); - xor_regs_2_(bytes, p1, p2); - kernel_vector_end(); -} - -static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1, - const unsigned long *__restrict p2, - const unsigned long *__restrict p3) -{ - kernel_vector_begin(); - xor_regs_3_(bytes, p1, p2, p3); - kernel_vector_end(); -} - -static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1, - const unsigned long *__restrict p2, - const unsigned long *__restrict p3, - const unsigned long *__restrict p4) -{ - kernel_vector_begin(); - xor_regs_4_(bytes, p1, p2, p3, p4); - kernel_vector_end(); -} - -static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1, - const unsigned long *__restrict p2, - const unsigned long *__restrict p3, - const unsigned long *__restrict p4, - const unsigned long *__restrict p5) -{ - kernel_vector_begin(); - xor_regs_5_(bytes, p1, p2, p3, p4, p5); - kernel_vector_end(); -} - -static struct xor_block_template xor_block_rvv = { - .name = "rvv", - .do_2 = xor_vector_2, - .do_3 = xor_vector_3, - .do_4 = xor_vector_4, - .do_5 = xor_vector_5 -}; - -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ - do { \ - xor_speed(&xor_block_8regs); \ - xor_speed(&xor_block_32regs); \ - if (has_vector()) { \ - xor_speed(&xor_block_rvv);\ - } \ - } while (0) -#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index bbc031124974..e220c35764eb 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -16,5 +16,4 @@ lib-$(CONFIG_MMU) += uaccess.o lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o -lib-$(CONFIG_RISCV_ISA_V) += xor.o lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o diff --git a/arch/s390/include/asm/xor.h b/arch/s390/include/asm/xor.h deleted file mode 100644 index 857d6759b67f..000000000000 --- a/arch/s390/include/asm/xor.h +++ /dev/null @@ -1,21 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * Optimited xor routines - * - * Copyright IBM Corp. 2016 - * Author(s): Martin Schwidefsky - */ -#ifndef _ASM_S390_XOR_H -#define _ASM_S390_XOR_H - -extern struct xor_block_template xor_block_xc; - -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ -do { \ - xor_speed(&xor_block_xc); \ -} while (0) - -#define XOR_SELECT_TEMPLATE(FASTEST) (&xor_block_xc) - -#endif /* _ASM_S390_XOR_H */ diff --git a/arch/s390/lib/Makefile b/arch/s390/lib/Makefile index f43f897d3fc0..2bf47204f6ab 100644 --- a/arch/s390/lib/Makefile +++ b/arch/s390/lib/Makefile @@ -5,7 +5,7 @@ lib-y += delay.o string.o uaccess.o find.o spinlock.o tishift.o lib-y += csum-partial.o -obj-y += mem.o xor.o +obj-y += mem.o lib-$(CONFIG_KPROBES) += probes.o lib-$(CONFIG_UPROBES) += probes.o obj-$(CONFIG_S390_KPROBES_SANITY_TEST) += test_kprobes_s390.o diff --git a/arch/sparc/include/asm/asm-prototypes.h b/arch/sparc/include/asm/asm-prototypes.h index 08810808ca6d..bbd1a8afaabf 100644 --- a/arch/sparc/include/asm/asm-prototypes.h +++ b/arch/sparc/include/asm/asm-prototypes.h @@ -14,7 +14,6 @@ #include #include #include -#include void *__memscan_zero(void *, size_t); void *__memscan_generic(void *, int, size_t); diff --git a/arch/sparc/include/asm/xor.h b/arch/sparc/include/asm/xor.h deleted file mode 100644 index f4c651e203c4..000000000000 --- a/arch/sparc/include/asm/xor.h +++ /dev/null @@ -1,9 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef ___ASM_SPARC_XOR_H -#define ___ASM_SPARC_XOR_H -#if defined(__sparc__) && defined(__arch64__) -#include -#else -#include -#endif -#endif diff --git a/arch/sparc/lib/Makefile b/arch/sparc/lib/Makefile index 783bdec0d7be..dd10cdd6f062 100644 --- a/arch/sparc/lib/Makefile +++ b/arch/sparc/lib/Makefile @@ -48,7 +48,7 @@ lib-$(CONFIG_SPARC64) += GENmemcpy.o GENcopy_from_user.o GENcopy_to_user.o lib-$(CONFIG_SPARC64) += GENpatch.o GENpage.o GENbzero.o lib-$(CONFIG_SPARC64) += copy_in_user.o memmove.o -lib-$(CONFIG_SPARC64) += mcount.o ipcsum.o xor.o hweight.o ffs.o +lib-$(CONFIG_SPARC64) += mcount.o ipcsum.o hweight.o ffs.o obj-$(CONFIG_SPARC64) += iomap.o obj-$(CONFIG_SPARC32) += atomic32.o diff --git a/arch/um/include/asm/xor.h b/arch/um/include/asm/xor.h deleted file mode 100644 index 647fae200c5d..000000000000 --- a/arch/um/include/asm/xor.h +++ /dev/null @@ -1,24 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_UM_XOR_H -#define _ASM_UM_XOR_H - -#ifdef CONFIG_64BIT -#undef CONFIG_X86_32 -#define TT_CPU_INF_XOR_DEFAULT (AVX_SELECT(&xor_block_sse_pf64)) -#else -#define CONFIG_X86_32 1 -#define TT_CPU_INF_XOR_DEFAULT (AVX_SELECT(&xor_block_8regs)) -#endif - -#include -#include <../../x86/include/asm/xor.h> -#include - -#ifdef CONFIG_UML_TIME_TRAVEL_SUPPORT -#undef XOR_SELECT_TEMPLATE -/* pick an arbitrary one - measuring isn't possible with inf-cpu */ -#define XOR_SELECT_TEMPLATE(x) \ - (time_travel_mode == TT_MODE_INFCPU ? TT_CPU_INF_XOR_DEFAULT : x) -#endif - -#endif diff --git a/arch/x86/include/asm/xor_64.h b/arch/x86/include/asm/xor_64.h deleted file mode 100644 index 0307e4ec5044..000000000000 --- a/arch/x86/include/asm/xor_64.h +++ /dev/null @@ -1,28 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_X86_XOR_64_H -#define _ASM_X86_XOR_64_H - -static struct xor_block_template xor_block_sse = { - .name = "generic_sse", - .do_2 = xor_sse_2, - .do_3 = xor_sse_3, - .do_4 = xor_sse_4, - .do_5 = xor_sse_5, -}; - - -/* Also try the AVX routines */ -#include - -/* We force the use of the SSE xor block because it can write around L2. - We may also be able to load into the L1 only depending on how the cpu - deals with a load to a line that is being prefetched. */ -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ -do { \ - AVX_XOR_SPEED; \ - xor_speed(&xor_block_sse_pf64); \ - xor_speed(&xor_block_sse); \ -} while (0) - -#endif /* _ASM_X86_XOR_64_H */ diff --git a/arch/x86/kernel/kexec-bzimage64.c b/arch/x86/kernel/kexec-bzimage64.c index 5630c7dca1f3..7e980ea49d8d 100644 --- a/arch/x86/kernel/kexec-bzimage64.c +++ b/arch/x86/kernel/kexec-bzimage64.c @@ -525,12 +525,8 @@ static void *bzImage64_load(struct kimage *image, char *kernel, if (ret) return ERR_PTR(ret); ret = crash_load_dm_crypt_keys(image); - if (ret == -ENOENT) { - kexec_dprintk("No dm crypt key to load\n"); - } else if (ret) { - pr_err("Failed to load dm crypt keys\n"); + if (ret) return ERR_PTR(ret); - } if (image->dm_crypt_keys_addr && cmdline_len + MAX_ELFCOREHDR_STR_LEN + MAX_DMCRYPTKEYS_STR_LEN > header->cmdline_size) { diff --git a/crypto/Kconfig b/crypto/Kconfig index b54a1bef6ade..103d1f58cb7c 100644 --- a/crypto/Kconfig +++ b/crypto/Kconfig @@ -2,8 +2,6 @@ # # Generic algorithms support # -config XOR_BLOCKS - tristate # # async_tx api: hardware offloaded memory transfer/transform support diff --git a/crypto/Makefile b/crypto/Makefile index d9bec7c6dc41..162242593c7c 100644 --- a/crypto/Makefile +++ b/crypto/Makefile @@ -193,7 +193,6 @@ obj-$(CONFIG_CRYPTO_ECRDSA) += ecrdsa_generic.o # # generic algorithms and the async_tx api # -obj-$(CONFIG_XOR_BLOCKS) += xor.o obj-$(CONFIG_ASYNC_CORE) += async_tx/ obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys/ crypto_simd-y := simd.o diff --git a/crypto/async_tx/async_xor.c b/crypto/async_tx/async_xor.c index 2c499654a36c..84458375b202 100644 --- a/crypto/async_tx/async_xor.c +++ b/crypto/async_tx/async_xor.c @@ -103,7 +103,6 @@ do_sync_xor_offs(struct page *dest, unsigned int offset, { int i; int xor_src_cnt = 0; - int src_off = 0; void *dest_buf; void **srcs; @@ -117,23 +116,12 @@ do_sync_xor_offs(struct page *dest, unsigned int offset, if (src_list[i]) srcs[xor_src_cnt++] = page_address(src_list[i]) + (src_offs ? src_offs[i] : offset); - src_cnt = xor_src_cnt; + /* set destination address */ dest_buf = page_address(dest) + offset; - if (submit->flags & ASYNC_TX_XOR_ZERO_DST) memset(dest_buf, 0, len); - - while (src_cnt > 0) { - /* process up to 'MAX_XOR_BLOCKS' sources */ - xor_src_cnt = min(src_cnt, MAX_XOR_BLOCKS); - xor_blocks(xor_src_cnt, len, dest_buf, &srcs[src_off]); - - /* drop completed sources */ - src_cnt -= xor_src_cnt; - src_off += xor_src_cnt; - } - + xor_gen(dest_buf, srcs, xor_src_cnt, len); async_tx_sync_epilog(submit); } @@ -168,11 +156,10 @@ dma_xor_aligned_offsets(struct dma_device *device, unsigned int offset, * * honored flags: ASYNC_TX_ACK, ASYNC_TX_XOR_ZERO_DST, ASYNC_TX_XOR_DROP_DST * - * xor_blocks always uses the dest as a source so the - * ASYNC_TX_XOR_ZERO_DST flag must be set to not include dest data in - * the calculation. The assumption with dma engines is that they only - * use the destination buffer as a source when it is explicitly specified - * in the source list. + * xor_gen always uses the dest as a source so the ASYNC_TX_XOR_ZERO_DST flag + * must be set to not include dest data in the calculation. The assumption with + * dma engines is that they only use the destination buffer as a source when it + * is explicitly specified in the source list. * * src_list note: if the dest is also a source it must be at index zero. * The contents of this array will be overwritten if a scribble region @@ -259,11 +246,10 @@ EXPORT_SYMBOL_GPL(async_xor_offs); * * honored flags: ASYNC_TX_ACK, ASYNC_TX_XOR_ZERO_DST, ASYNC_TX_XOR_DROP_DST * - * xor_blocks always uses the dest as a source so the - * ASYNC_TX_XOR_ZERO_DST flag must be set to not include dest data in - * the calculation. The assumption with dma engines is that they only - * use the destination buffer as a source when it is explicitly specified - * in the source list. + * xor_gen always uses the dest as a source so the ASYNC_TX_XOR_ZERO_DST flag + * must be set to not include dest data in the calculation. The assumption with + * dma engines is that they only use the destination buffer as a source when it + * is explicitly specified in the source list. * * src_list note: if the dest is also a source it must be at index zero. * The contents of this array will be overwritten if a scribble region diff --git a/crypto/xor.c b/crypto/xor.c deleted file mode 100644 index f39621a57bb3..000000000000 --- a/crypto/xor.c +++ /dev/null @@ -1,174 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * xor.c : Multiple Devices driver for Linux - * - * Copyright (C) 1996, 1997, 1998, 1999, 2000, - * Ingo Molnar, Matti Aarnio, Jakub Jelinek, Richard Henderson. - * - * Dispatch optimized RAID-5 checksumming functions. - */ - -#define BH_TRACE 0 -#include -#include -#include -#include -#include -#include - -#ifndef XOR_SELECT_TEMPLATE -#define XOR_SELECT_TEMPLATE(x) (x) -#endif - -/* The xor routines to use. */ -static struct xor_block_template *active_template; - -void -xor_blocks(unsigned int src_count, unsigned int bytes, void *dest, void **srcs) -{ - unsigned long *p1, *p2, *p3, *p4; - - p1 = (unsigned long *) srcs[0]; - if (src_count == 1) { - active_template->do_2(bytes, dest, p1); - return; - } - - p2 = (unsigned long *) srcs[1]; - if (src_count == 2) { - active_template->do_3(bytes, dest, p1, p2); - return; - } - - p3 = (unsigned long *) srcs[2]; - if (src_count == 3) { - active_template->do_4(bytes, dest, p1, p2, p3); - return; - } - - p4 = (unsigned long *) srcs[3]; - active_template->do_5(bytes, dest, p1, p2, p3, p4); -} -EXPORT_SYMBOL(xor_blocks); - -/* Set of all registered templates. */ -static struct xor_block_template *__initdata template_list; - -#ifndef MODULE -static void __init do_xor_register(struct xor_block_template *tmpl) -{ - tmpl->next = template_list; - template_list = tmpl; -} - -static int __init register_xor_blocks(void) -{ - active_template = XOR_SELECT_TEMPLATE(NULL); - - if (!active_template) { -#define xor_speed do_xor_register - // register all the templates and pick the first as the default - XOR_TRY_TEMPLATES; -#undef xor_speed - active_template = template_list; - } - return 0; -} -#endif - -#define BENCH_SIZE 4096 -#define REPS 800U - -static void __init -do_xor_speed(struct xor_block_template *tmpl, void *b1, void *b2) -{ - int speed; - unsigned long reps; - ktime_t min, start, t0; - - tmpl->next = template_list; - template_list = tmpl; - - preempt_disable(); - - reps = 0; - t0 = ktime_get(); - /* delay start until time has advanced */ - while ((start = ktime_get()) == t0) - cpu_relax(); - do { - mb(); /* prevent loop optimization */ - tmpl->do_2(BENCH_SIZE, b1, b2); - mb(); - } while (reps++ < REPS || (t0 = ktime_get()) == start); - min = ktime_sub(t0, start); - - preempt_enable(); - - // bytes/ns == GB/s, multiply by 1000 to get MB/s [not MiB/s] - speed = (1000 * reps * BENCH_SIZE) / (unsigned int)ktime_to_ns(min); - tmpl->speed = speed; - - pr_info(" %-16s: %5d MB/sec\n", tmpl->name, speed); -} - -static int __init -calibrate_xor_blocks(void) -{ - void *b1, *b2; - struct xor_block_template *f, *fastest; - - fastest = XOR_SELECT_TEMPLATE(NULL); - - if (fastest) { - printk(KERN_INFO "xor: automatically using best " - "checksumming function %-10s\n", - fastest->name); - goto out; - } - - b1 = (void *) __get_free_pages(GFP_KERNEL, 2); - if (!b1) { - printk(KERN_WARNING "xor: Yikes! No memory available.\n"); - return -ENOMEM; - } - b2 = b1 + 2*PAGE_SIZE + BENCH_SIZE; - - /* - * If this arch/cpu has a short-circuited selection, don't loop through - * all the possible functions, just test the best one - */ - -#define xor_speed(templ) do_xor_speed((templ), b1, b2) - - printk(KERN_INFO "xor: measuring software checksum speed\n"); - template_list = NULL; - XOR_TRY_TEMPLATES; - fastest = template_list; - for (f = fastest; f; f = f->next) - if (f->speed > fastest->speed) - fastest = f; - - pr_info("xor: using function: %s (%d MB/sec)\n", - fastest->name, fastest->speed); - -#undef xor_speed - - free_pages((unsigned long)b1, 2); -out: - active_template = fastest; - return 0; -} - -static __exit void xor_exit(void) { } - -MODULE_DESCRIPTION("RAID-5 checksumming functions"); -MODULE_LICENSE("GPL"); - -#ifndef MODULE -/* when built-in xor.o must initialize before drivers/md/md.o */ -core_initcall(register_xor_blocks); -#endif - -module_init(calibrate_xor_blocks); -module_exit(xor_exit); diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c index 331646d667b9..2967e4aff807 100644 --- a/drivers/of/fdt.c +++ b/drivers/of/fdt.c @@ -866,6 +866,26 @@ static void __init early_init_dt_check_for_elfcorehdr(unsigned long node) elfcorehdr_addr, elfcorehdr_size); } +static void __init early_init_dt_check_for_dmcryptkeys(unsigned long node) +{ + const char *prop_name = "linux,dmcryptkeys"; + const __be32 *prop; + + if (!IS_ENABLED(CONFIG_CRASH_DM_CRYPT)) + return; + + pr_debug("Looking for dmcryptkeys property... "); + + prop = of_get_flat_dt_prop(node, prop_name, NULL); + if (!prop) + return; + + dm_crypt_keys_addr = dt_mem_next_cell(dt_root_addr_cells, &prop); + + /* Property only accessible to crash dump kernel */ + fdt_delprop(initial_boot_params, node, prop_name); +} + static unsigned long chosen_node_offset = -FDT_ERR_NOTFOUND; /* @@ -1097,6 +1117,7 @@ int __init early_init_dt_scan_chosen(char *cmdline) early_init_dt_check_for_initrd(node); early_init_dt_check_for_elfcorehdr(node); + early_init_dt_check_for_dmcryptkeys(node); rng_seed = of_get_flat_dt_prop(node, "rng-seed", &l); if (rng_seed && l > 0) { diff --git a/drivers/of/kexec.c b/drivers/of/kexec.c index c4cf3552c018..fbd253f0d3c5 100644 --- a/drivers/of/kexec.c +++ b/drivers/of/kexec.c @@ -423,6 +423,25 @@ void *of_kexec_alloc_and_setup_fdt(const struct kimage *image, if (ret) goto out; + if (image->dm_crypt_keys_addr != 0) { + ret = fdt_appendprop_addrrange(fdt, 0, chosen_node, + "linux,dmcryptkeys", + image->dm_crypt_keys_addr, + image->dm_crypt_keys_sz); + + if (ret) + goto out; + + /* + * Avoid dmcryptkeys from being stomped on in kdump kernel by + * setting up memory reserve map. + */ + ret = fdt_add_mem_rsv(fdt, image->dm_crypt_keys_addr, + image->dm_crypt_keys_sz); + if (ret) + goto out; + } + #ifdef CONFIG_CRASH_DUMP /* add linux,usable-memory-range */ ret = fdt_appendprop_addrrange(fdt, 0, chosen_node, diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c index e31d57d6ab1e..baebd9f733e9 100644 --- a/fs/btrfs/raid56.c +++ b/fs/btrfs/raid56.c @@ -617,26 +617,6 @@ static void cache_rbio(struct btrfs_raid_bio *rbio) spin_unlock(&table->cache_lock); } -/* - * helper function to run the xor_blocks api. It is only - * able to do MAX_XOR_BLOCKS at a time, so we need to - * loop through. - */ -static void run_xor(void **pages, int src_cnt, ssize_t len) -{ - int src_off = 0; - int xor_src_cnt = 0; - void *dest = pages[src_cnt]; - - while(src_cnt > 0) { - xor_src_cnt = min(src_cnt, MAX_XOR_BLOCKS); - xor_blocks(xor_src_cnt, len, dest, pages + src_off); - - src_cnt -= xor_src_cnt; - src_off += xor_src_cnt; - } -} - /* * Returns true if the bio list inside this rbio covers an entire stripe (no * rmw required). @@ -1434,7 +1414,8 @@ static void generate_pq_vertical_step(struct btrfs_raid_bio *rbio, unsigned int } else { /* raid5 */ memcpy(pointers[rbio->nr_data], pointers[0], step); - run_xor(pointers + 1, rbio->nr_data - 1, step); + xor_gen(pointers[rbio->nr_data], pointers + 1, rbio->nr_data - 1, + step); } for (stripe = stripe - 1; stripe >= 0; stripe--) kunmap_local(pointers[stripe]); @@ -2029,7 +2010,7 @@ pstripe: pointers[rbio->nr_data - 1] = p; /* Xor in the rest */ - run_xor(pointers, rbio->nr_data - 1, step); + xor_gen(p, pointers, rbio->nr_data - 1, step); } cleanup: @@ -2667,7 +2648,7 @@ static bool verify_one_parity_step(struct btrfs_raid_bio *rbio, } else { /* RAID5. */ memcpy(pointers[nr_data], pointers[0], step); - run_xor(pointers + 1, nr_data - 1, step); + xor_gen(pointers[nr_data], pointers + 1, nr_data - 1, step); } /* Check scrubbing parity and repair it. */ diff --git a/fs/ocfs2/alloc.c b/fs/ocfs2/alloc.c index d40f5d205bce..6e5fd3f12a84 100644 --- a/fs/ocfs2/alloc.c +++ b/fs/ocfs2/alloc.c @@ -917,11 +917,32 @@ static int ocfs2_validate_extent_block(struct super_block *sb, goto bail; } - if (le32_to_cpu(eb->h_fs_generation) != OCFS2_SB(sb)->fs_generation) + if (le32_to_cpu(eb->h_fs_generation) != OCFS2_SB(sb)->fs_generation) { rc = ocfs2_error(sb, "Extent block #%llu has an invalid h_fs_generation of #%u\n", (unsigned long long)bh->b_blocknr, le32_to_cpu(eb->h_fs_generation)); + goto bail; + } + + if (le16_to_cpu(eb->h_list.l_count) != ocfs2_extent_recs_per_eb(sb)) { + rc = ocfs2_error(sb, + "Extent block #%llu has invalid l_count %u (expected %u)\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(eb->h_list.l_count), + ocfs2_extent_recs_per_eb(sb)); + goto bail; + } + + if (le16_to_cpu(eb->h_list.l_next_free_rec) > le16_to_cpu(eb->h_list.l_count)) { + rc = ocfs2_error(sb, + "Extent block #%llu has invalid l_next_free_rec %u (l_count %u)\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(eb->h_list.l_next_free_rec), + le16_to_cpu(eb->h_list.l_count)); + goto bail; + } + bail: return rc; } @@ -1857,18 +1878,6 @@ static int __ocfs2_find_path(struct ocfs2_caching_info *ci, eb = (struct ocfs2_extent_block *) bh->b_data; el = &eb->h_list; - if (le16_to_cpu(el->l_next_free_rec) > - le16_to_cpu(el->l_count)) { - ocfs2_error(ocfs2_metadata_cache_get_super(ci), - "Owner %llu has bad count in extent list at block %llu (next free=%u, count=%u)\n", - (unsigned long long)ocfs2_metadata_cache_owner(ci), - (unsigned long long)bh->b_blocknr, - le16_to_cpu(el->l_next_free_rec), - le16_to_cpu(el->l_count)); - ret = -EROFS; - goto out; - } - if (func) func(data, bh); } diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c index c7ad912ec7a0..6ec198bdab12 100644 --- a/fs/ocfs2/aops.c +++ b/fs/ocfs2/aops.c @@ -37,6 +37,8 @@ #include "namei.h" #include "sysfile.h" +#define OCFS2_DIO_MARK_EXTENT_BATCH 200 + static int ocfs2_symlink_get_block(struct inode *inode, sector_t iblock, struct buffer_head *bh_result, int create) { @@ -2277,7 +2279,7 @@ static int ocfs2_dio_end_io_write(struct inode *inode, struct ocfs2_alloc_context *meta_ac = NULL; handle_t *handle = NULL; loff_t end = offset + bytes; - int ret = 0, credits = 0; + int ret = 0, credits = 0, batch = 0; ocfs2_init_dealloc_ctxt(&dealloc); @@ -2295,19 +2297,6 @@ static int ocfs2_dio_end_io_write(struct inode *inode, } down_write(&oi->ip_alloc_sem); - - /* Delete orphan before acquire i_rwsem. */ - if (dwc->dw_orphaned) { - BUG_ON(dwc->dw_writer_pid != task_pid_nr(current)); - - end = end > i_size_read(inode) ? end : 0; - - ret = ocfs2_del_inode_from_orphan(osb, inode, di_bh, - !!end, end); - if (ret < 0) - mlog_errno(ret); - } - di = (struct ocfs2_dinode *)di_bh->b_data; ocfs2_init_dinode_extent_tree(&et, INODE_CACHE(inode), di_bh); @@ -2327,24 +2316,25 @@ static int ocfs2_dio_end_io_write(struct inode *inode, credits = ocfs2_calc_extend_credits(inode->i_sb, &di->id2.i_list); - handle = ocfs2_start_trans(osb, credits); - if (IS_ERR(handle)) { - ret = PTR_ERR(handle); - mlog_errno(ret); - goto unlock; - } - ret = ocfs2_journal_access_di(handle, INODE_CACHE(inode), di_bh, - OCFS2_JOURNAL_ACCESS_WRITE); - if (ret) { - mlog_errno(ret); - goto commit; - } - list_for_each_entry(ue, &dwc->dw_zero_list, ue_node) { + if (!handle) { + handle = ocfs2_start_trans(osb, credits); + if (IS_ERR(handle)) { + ret = PTR_ERR(handle); + mlog_errno(ret); + goto unlock; + } + ret = ocfs2_journal_access_di(handle, INODE_CACHE(inode), di_bh, + OCFS2_JOURNAL_ACCESS_WRITE); + if (ret) { + mlog_errno(ret); + goto commit; + } + } ret = ocfs2_assure_trans_credits(handle, credits); if (ret < 0) { mlog_errno(ret); - break; + goto commit; } ret = ocfs2_mark_extent_written(inode, &et, handle, ue->ue_cpos, 1, @@ -2352,19 +2342,44 @@ static int ocfs2_dio_end_io_write(struct inode *inode, meta_ac, &dealloc); if (ret < 0) { mlog_errno(ret); - break; + goto commit; + } + + if (++batch == OCFS2_DIO_MARK_EXTENT_BATCH) { + ocfs2_commit_trans(osb, handle); + handle = NULL; + batch = 0; } } if (end > i_size_read(inode)) { + if (!handle) { + handle = ocfs2_start_trans(osb, credits); + if (IS_ERR(handle)) { + ret = PTR_ERR(handle); + mlog_errno(ret); + goto unlock; + } + } ret = ocfs2_set_inode_size(handle, inode, di_bh, end); if (ret < 0) mlog_errno(ret); } + commit: - ocfs2_commit_trans(osb, handle); + if (handle) + ocfs2_commit_trans(osb, handle); unlock: up_write(&oi->ip_alloc_sem); + + /* everything looks good, let's start the cleanup */ + if (!ret && dwc->dw_orphaned) { + BUG_ON(dwc->dw_writer_pid != task_pid_nr(current)); + + ret = ocfs2_del_inode_from_orphan(osb, inode, di_bh, 0, 0); + if (ret < 0) + mlog_errno(ret); + } ocfs2_inode_unlock(inode, 1); brelse(di_bh); out: diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c index fe1949578336..d12784aaaa4b 100644 --- a/fs/ocfs2/cluster/heartbeat.c +++ b/fs/ocfs2/cluster/heartbeat.c @@ -1488,33 +1488,45 @@ static struct o2hb_region *to_o2hb_region(struct config_item *item) return item ? container_of(item, struct o2hb_region, hr_item) : NULL; } -/* drop_item only drops its ref after killing the thread, nothing should - * be using the region anymore. this has to clean up any state that - * attributes might have built up. */ -static void o2hb_region_release(struct config_item *item) +static void o2hb_unmap_slot_data(struct o2hb_region *reg) { int i; struct page *page; - struct o2hb_region *reg = to_o2hb_region(item); - - mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg_bdev(reg)); - - kfree(reg->hr_tmp_block); if (reg->hr_slot_data) { for (i = 0; i < reg->hr_num_pages; i++) { page = reg->hr_slot_data[i]; - if (page) + if (page) { __free_page(page); + reg->hr_slot_data[i] = NULL; + } } kfree(reg->hr_slot_data); + reg->hr_slot_data = NULL; } + kfree(reg->hr_slots); + reg->hr_slots = NULL; + + kfree(reg->hr_tmp_block); + reg->hr_tmp_block = NULL; +} + +/* drop_item only drops its ref after killing the thread, nothing should + * be using the region anymore. this has to clean up any state that + * attributes might have built up. + */ +static void o2hb_region_release(struct config_item *item) +{ + struct o2hb_region *reg = to_o2hb_region(item); + + mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg_bdev(reg)); + + o2hb_unmap_slot_data(reg); + if (reg->hr_bdev_file) fput(reg->hr_bdev_file); - kfree(reg->hr_slots); - debugfs_remove_recursive(reg->hr_debug_dir); kfree(reg->hr_db_livenodes); kfree(reg->hr_db_regnum); @@ -1667,6 +1679,7 @@ static void o2hb_init_region_params(struct o2hb_region *reg) static int o2hb_map_slot_data(struct o2hb_region *reg) { int i, j; + int ret = -ENOMEM; unsigned int last_slot; unsigned int spp = reg->hr_slots_per_page; struct page *page; @@ -1674,14 +1687,14 @@ static int o2hb_map_slot_data(struct o2hb_region *reg) struct o2hb_disk_slot *slot; reg->hr_tmp_block = kmalloc(reg->hr_block_bytes, GFP_KERNEL); - if (reg->hr_tmp_block == NULL) - return -ENOMEM; + if (!reg->hr_tmp_block) + goto out; reg->hr_slots = kzalloc_objs(struct o2hb_disk_slot, reg->hr_blocks); - if (reg->hr_slots == NULL) - return -ENOMEM; + if (!reg->hr_slots) + goto out; - for(i = 0; i < reg->hr_blocks; i++) { + for (i = 0; i < reg->hr_blocks; i++) { slot = ®->hr_slots[i]; slot->ds_node_num = i; INIT_LIST_HEAD(&slot->ds_live_item); @@ -1695,12 +1708,12 @@ static int o2hb_map_slot_data(struct o2hb_region *reg) reg->hr_slot_data = kzalloc_objs(struct page *, reg->hr_num_pages); if (!reg->hr_slot_data) - return -ENOMEM; + goto out; - for(i = 0; i < reg->hr_num_pages; i++) { + for (i = 0; i < reg->hr_num_pages; i++) { page = alloc_page(GFP_KERNEL); if (!page) - return -ENOMEM; + goto out; reg->hr_slot_data[i] = page; @@ -1720,6 +1733,10 @@ static int o2hb_map_slot_data(struct o2hb_region *reg) } return 0; + +out: + o2hb_unmap_slot_data(reg); + return ret; } /* Read in all the slots available and populate the tracking @@ -1809,9 +1826,11 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, "blocksize %u incorrect for device, expected %d", reg->hr_block_bytes, sectsize); ret = -EINVAL; - goto out3; + goto out; } + reg->hr_aborted_start = 0; + reg->hr_node_deleted = 0; o2hb_init_region_params(reg); /* Generation of zero is invalid */ @@ -1823,13 +1842,13 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, ret = o2hb_map_slot_data(reg); if (ret) { mlog_errno(ret); - goto out3; + goto out; } ret = o2hb_populate_slot_data(reg); if (ret) { mlog_errno(ret); - goto out3; + goto out; } INIT_DELAYED_WORK(®->hr_write_timeout_work, o2hb_write_timeout); @@ -1860,7 +1879,7 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, if (IS_ERR(hb_task)) { ret = PTR_ERR(hb_task); mlog_errno(ret); - goto out3; + goto out; } spin_lock(&o2hb_live_lock); @@ -1877,12 +1896,12 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, if (reg->hr_aborted_start) { ret = -EIO; - goto out3; + goto out; } if (reg->hr_node_deleted) { ret = -EINVAL; - goto out3; + goto out; } /* Ok, we were woken. Make sure it wasn't by drop_item() */ @@ -1901,8 +1920,18 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, printk(KERN_NOTICE "o2hb: Heartbeat started on region %s (%pg)\n", config_item_name(®->hr_item), reg_bdev(reg)); -out3: +out: if (ret < 0) { + spin_lock(&o2hb_live_lock); + hb_task = reg->hr_task; + reg->hr_task = NULL; + spin_unlock(&o2hb_live_lock); + + if (hb_task) + kthread_stop(hb_task); + + o2hb_unmap_slot_data(reg); + fput(reg->hr_bdev_file); reg->hr_bdev_file = NULL; } diff --git a/fs/ocfs2/dir.c b/fs/ocfs2/dir.c index b82fe4431eb1..8e6b03238327 100644 --- a/fs/ocfs2/dir.c +++ b/fs/ocfs2/dir.c @@ -593,7 +593,7 @@ static int ocfs2_validate_dx_root(struct super_block *sb, mlog(ML_ERROR, "Checksum failed for dir index root block %llu\n", (unsigned long long)bh->b_blocknr); - return ret; + goto bail; } if (!OCFS2_IS_VALID_DX_ROOT(dx_root)) { @@ -601,8 +601,32 @@ static int ocfs2_validate_dx_root(struct super_block *sb, "Dir Index Root # %llu has bad signature %.*s\n", (unsigned long long)le64_to_cpu(dx_root->dr_blkno), 7, dx_root->dr_signature); + goto bail; } + if (!(dx_root->dr_flags & OCFS2_DX_FLAG_INLINE)) { + struct ocfs2_extent_list *el = &dx_root->dr_list; + + if (le16_to_cpu(el->l_count) != ocfs2_extent_recs_per_dx_root(sb)) { + ret = ocfs2_error(sb, + "Dir Index Root # %llu has invalid l_count %u (expected %u)\n", + (unsigned long long)le64_to_cpu(dx_root->dr_blkno), + le16_to_cpu(el->l_count), + ocfs2_extent_recs_per_dx_root(sb)); + goto bail; + } + + if (le16_to_cpu(el->l_next_free_rec) > le16_to_cpu(el->l_count)) { + ret = ocfs2_error(sb, + "Dir Index Root # %llu has invalid l_next_free_rec %u (l_count %u)\n", + (unsigned long long)le64_to_cpu(dx_root->dr_blkno), + le16_to_cpu(el->l_next_free_rec), + le16_to_cpu(el->l_count)); + goto bail; + } + } + +bail: return ret; } @@ -791,14 +815,6 @@ static int ocfs2_dx_dir_lookup_rec(struct inode *inode, struct ocfs2_extent_block *eb; struct ocfs2_extent_rec *rec = NULL; - if (le16_to_cpu(el->l_count) != - ocfs2_extent_recs_per_dx_root(inode->i_sb)) { - ret = ocfs2_error(inode->i_sb, - "Inode %llu has invalid extent list length %u\n", - inode->i_ino, le16_to_cpu(el->l_count)); - goto out; - } - if (el->l_tree_depth) { ret = ocfs2_find_leaf(INODE_CACHE(inode), el, major_hash, &eb_bh); @@ -819,14 +835,6 @@ static int ocfs2_dx_dir_lookup_rec(struct inode *inode, } } - if (le16_to_cpu(el->l_next_free_rec) == 0) { - ret = ocfs2_error(inode->i_sb, - "Inode %llu has empty extent list at depth %u\n", - inode->i_ino, - le16_to_cpu(el->l_tree_depth)); - goto out; - } - found = 0; for (i = le16_to_cpu(el->l_next_free_rec) - 1; i >= 0; i--) { rec = &el->l_recs[i]; @@ -839,10 +847,9 @@ static int ocfs2_dx_dir_lookup_rec(struct inode *inode, if (!found) { ret = ocfs2_error(inode->i_sb, - "Inode %llu has bad extent record (%u, %u, 0) in btree\n", - inode->i_ino, - le32_to_cpu(rec->e_cpos), - ocfs2_rec_clusters(el, rec)); + "Inode %llu has no extent record for hash %u in btree (next_free_rec %u)\n", + inode->i_ino, major_hash, + le16_to_cpu(el->l_next_free_rec)); goto out; } diff --git a/fs/ocfs2/dlm/dlmdomain.c b/fs/ocfs2/dlm/dlmdomain.c index 70ca79e4bdc3..dc9da9133c8e 100644 --- a/fs/ocfs2/dlm/dlmdomain.c +++ b/fs/ocfs2/dlm/dlmdomain.c @@ -980,6 +980,14 @@ static int dlm_match_regions(struct dlm_ctxt *dlm, goto bail; } + if (qr->qr_numregions > O2NM_MAX_REGIONS) { + mlog(ML_ERROR, "Domain %s: Joining node %d has invalid " + "number of heartbeat regions %u\n", + qr->qr_domain, qr->qr_node, qr->qr_numregions); + status = -EINVAL; + goto bail; + } + r = remote; for (i = 0; i < qr->qr_numregions; ++i) { mlog(0, "Region %.*s\n", O2HB_MAX_REGION_NAME_LEN, r); @@ -994,7 +1002,7 @@ static int dlm_match_regions(struct dlm_ctxt *dlm, for (i = 0; i < localnr; ++i) { foundit = 0; r = remote; - for (j = 0; j <= qr->qr_numregions; ++j) { + for (j = 0; j < qr->qr_numregions; ++j) { if (!memcmp(l, r, O2HB_MAX_REGION_NAME_LEN)) { foundit = 1; break; diff --git a/fs/ocfs2/dlm/dlmmaster.c b/fs/ocfs2/dlm/dlmmaster.c index eb62724bbe9b..93eff38fdadd 100644 --- a/fs/ocfs2/dlm/dlmmaster.c +++ b/fs/ocfs2/dlm/dlmmaster.c @@ -930,7 +930,6 @@ redo_request: if (blocked) goto wait; - ret = -EINVAL; dlm_node_iter_init(mle->vote_map, &iter); while ((nodenum = dlm_node_iter_next(&iter)) >= 0) { ret = dlm_do_master_request(res, mle, nodenum); diff --git a/fs/ocfs2/ioctl.c b/fs/ocfs2/ioctl.c index bfed0fb35f9b..cbe59d231666 100644 --- a/fs/ocfs2/ioctl.c +++ b/fs/ocfs2/ioctl.c @@ -441,13 +441,16 @@ static int ocfs2_info_freefrag_scan_chain(struct ocfs2_super *osb, struct buffer_head *bh = NULL; struct ocfs2_group_desc *bg = NULL; - unsigned int max_bits, num_clusters; + unsigned int max_bits, max_bitmap_bits, num_clusters; unsigned int offset = 0, cluster, chunk; unsigned int chunk_free, last_chunksize = 0; if (!le32_to_cpu(rec->c_free)) goto bail; + max_bitmap_bits = 8 * ocfs2_group_bitmap_size(osb->sb, 0, + osb->s_feature_incompat); + do { if (!bg) blkno = le64_to_cpu(rec->c_blkno); @@ -479,6 +482,19 @@ static int ocfs2_info_freefrag_scan_chain(struct ocfs2_super *osb, continue; max_bits = le16_to_cpu(bg->bg_bits); + + /* + * Non-coherent scans read raw blocks and do not get the + * bg_bits validation from + * ocfs2_read_group_descriptor(). + */ + if (max_bits > max_bitmap_bits) { + mlog(ML_ERROR, + "Group desc #%llu has %u bits, max bitmap bits %u\n", + (unsigned long long)blkno, max_bits, max_bitmap_bits); + max_bits = max_bitmap_bits; + } + offset = 0; for (chunk = 0; chunk < chunks_in_group; chunk++) { diff --git a/fs/ocfs2/mmap.c b/fs/ocfs2/mmap.c index 50e2faf64c19..6c570157caf1 100644 --- a/fs/ocfs2/mmap.c +++ b/fs/ocfs2/mmap.c @@ -30,7 +30,8 @@ static vm_fault_t ocfs2_fault(struct vm_fault *vmf) { - struct vm_area_struct *vma = vmf->vma; + unsigned long long ip_blkno = + OCFS2_I(file_inode(vmf->vma->vm_file))->ip_blkno; sigset_t oldset; vm_fault_t ret; @@ -38,11 +39,9 @@ static vm_fault_t ocfs2_fault(struct vm_fault *vmf) ret = filemap_fault(vmf); ocfs2_unblock_signals(&oldset); - trace_ocfs2_fault(OCFS2_I(vma->vm_file->f_mapping->host)->ip_blkno, - vma, vmf->page, vmf->pgoff); + trace_ocfs2_fault(ip_blkno, vmf->page, vmf->pgoff); return ret; } - static vm_fault_t __ocfs2_page_mkwrite(struct file *file, struct buffer_head *di_bh, struct folio *folio) { diff --git a/fs/ocfs2/ocfs2_trace.h b/fs/ocfs2/ocfs2_trace.h index 4b32fb5658ad..6c2c97a9804f 100644 --- a/fs/ocfs2/ocfs2_trace.h +++ b/fs/ocfs2/ocfs2_trace.h @@ -1246,22 +1246,20 @@ TRACE_EVENT(ocfs2_write_end_inline, TRACE_EVENT(ocfs2_fault, TP_PROTO(unsigned long long ino, - void *area, void *page, unsigned long pgoff), - TP_ARGS(ino, area, page, pgoff), + void *page, unsigned long pgoff), + TP_ARGS(ino, page, pgoff), TP_STRUCT__entry( __field(unsigned long long, ino) - __field(void *, area) __field(void *, page) __field(unsigned long, pgoff) ), TP_fast_assign( __entry->ino = ino; - __entry->area = area; __entry->page = page; __entry->pgoff = pgoff; ), - TP_printk("%llu %p %p %lu", - __entry->ino, __entry->area, __entry->page, __entry->pgoff) + TP_printk("%llu %p %lu", + __entry->ino, __entry->page, __entry->pgoff) ); /* End of trace events for fs/ocfs2/mmap.c. */ diff --git a/fs/ocfs2/quota_global.c b/fs/ocfs2/quota_global.c index e85b1ccf81be..77b8f0363e94 100644 --- a/fs/ocfs2/quota_global.c +++ b/fs/ocfs2/quota_global.c @@ -311,11 +311,25 @@ int ocfs2_lock_global_qf(struct ocfs2_mem_dqinfo *oinfo, int ex) spin_unlock(&dq_data_lock); if (ex) { inode_lock(oinfo->dqi_gqinode); - down_write(&OCFS2_I(oinfo->dqi_gqinode)->ip_alloc_sem); + if (!down_write_trylock(&OCFS2_I(oinfo->dqi_gqinode)->ip_alloc_sem)) { + inode_unlock(oinfo->dqi_gqinode); + status = -EBUSY; + goto bail; + } } else { down_read(&OCFS2_I(oinfo->dqi_gqinode)->ip_alloc_sem); } return 0; + +bail: + /* does a similar job as ocfs2_unlock_global_qf */ + ocfs2_inode_unlock(oinfo->dqi_gqinode, ex); + brelse(oinfo->dqi_gqi_bh); + spin_lock(&dq_data_lock); + if (!--oinfo->dqi_gqi_count) + oinfo->dqi_gqi_bh = NULL; + spin_unlock(&dq_data_lock); + return status; } void ocfs2_unlock_global_qf(struct ocfs2_mem_dqinfo *oinfo, int ex) diff --git a/fs/ocfs2/quota_local.c b/fs/ocfs2/quota_local.c index 269b0f27d567..12cbb4fccda0 100644 --- a/fs/ocfs2/quota_local.c +++ b/fs/ocfs2/quota_local.c @@ -1224,7 +1224,9 @@ int ocfs2_create_local_dquot(struct dquot *dquot) int status; u64 pcount; - down_write(&OCFS2_I(lqinode)->ip_alloc_sem); + if (!down_write_trylock(&OCFS2_I(lqinode)->ip_alloc_sem)) + return -EBUSY; + chunk = ocfs2_find_free_entry(sb, type, &offset); if (!chunk) { chunk = ocfs2_extend_local_quota_file(sb, type, &offset); diff --git a/fs/ocfs2/resize.c b/fs/ocfs2/resize.c index ac3ec2c21119..6375d5035972 100644 --- a/fs/ocfs2/resize.c +++ b/fs/ocfs2/resize.c @@ -303,9 +303,13 @@ int ocfs2_group_extend(struct inode * inode, int new_clusters) fe = (struct ocfs2_dinode *)main_bm_bh->b_data; - /* main_bm_bh is validated by inode read inside ocfs2_inode_lock(), - * so any corruption is a code bug. */ - BUG_ON(!OCFS2_IS_VALID_DINODE(fe)); + /* JBD-managed buffers can bypass validation, so treat this as corruption. */ + if (!OCFS2_IS_VALID_DINODE(fe)) { + ret = ocfs2_error(main_bm_inode->i_sb, + "Invalid dinode #%llu\n", + (unsigned long long)OCFS2_I(main_bm_inode)->ip_blkno); + goto out_unlock; + } if (le16_to_cpu(fe->id2.i_chain.cl_cpg) != ocfs2_group_bitmap_size(osb->sb, 0, @@ -504,14 +508,14 @@ int ocfs2_group_add(struct inode *inode, struct ocfs2_new_group_input *input) goto out_unlock; } - ocfs2_set_new_buffer_uptodate(INODE_CACHE(inode), group_bh); - ret = ocfs2_verify_group_and_input(main_bm_inode, fe, input, group_bh); if (ret) { mlog_errno(ret); goto out_free_group_bh; } + ocfs2_set_new_buffer_uptodate(INODE_CACHE(main_bm_inode), group_bh); + trace_ocfs2_group_add((unsigned long long)input->group, input->chain, input->clusters, input->frees); @@ -519,7 +523,7 @@ int ocfs2_group_add(struct inode *inode, struct ocfs2_new_group_input *input) if (IS_ERR(handle)) { mlog_errno(PTR_ERR(handle)); ret = -EINVAL; - goto out_free_group_bh; + goto out_remove_cache; } cl_bpc = le16_to_cpu(fe->id2.i_chain.cl_bpc); @@ -573,9 +577,11 @@ int ocfs2_group_add(struct inode *inode, struct ocfs2_new_group_input *input) out_commit: ocfs2_commit_trans(osb, handle); -out_free_group_bh: +out_remove_cache: if (ret < 0) - ocfs2_remove_from_cache(INODE_CACHE(inode), group_bh); + ocfs2_remove_from_cache(INODE_CACHE(main_bm_inode), group_bh); + +out_free_group_bh: brelse(group_bh); out_unlock: diff --git a/fs/ocfs2/suballoc.c b/fs/ocfs2/suballoc.c index bb98bd51338e..d284e0e37252 100644 --- a/fs/ocfs2/suballoc.c +++ b/fs/ocfs2/suballoc.c @@ -197,6 +197,31 @@ static int ocfs2_validate_gd_self(struct super_block *sb, 8 * le16_to_cpu(gd->bg_size)); } + /* + * For discontiguous block groups, validate the on-disk extent list + * against the maximum number of extent records that can physically + * fit in a single block. + */ + if (ocfs2_gd_is_discontig(gd)) { + u16 max_recs = ocfs2_extent_recs_per_gd(sb); + u16 l_count = le16_to_cpu(gd->bg_list.l_count); + u16 l_next_free_rec = le16_to_cpu(gd->bg_list.l_next_free_rec); + + if (l_count != max_recs) { + do_error("Group descriptor #%llu bad discontig l_count %u expected %u\n", + (unsigned long long)bh->b_blocknr, + l_count, + max_recs); + } + + if (l_next_free_rec > l_count) { + do_error("Group descriptor #%llu bad discontig l_next_free_rec %u max %u\n", + (unsigned long long)bh->b_blocknr, + l_next_free_rec, + l_count); + } + } + return 0; } diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index d7c58fd7d438..b875f01c9756 100644 --- a/fs/ocfs2/super.c +++ b/fs/ocfs2/super.c @@ -2124,7 +2124,7 @@ static int ocfs2_initialize_super(struct super_block *sb, osb->osb_cluster_stack[0] = '\0'; } - get_random_bytes(&osb->s_next_generation, sizeof(u32)); + osb->s_next_generation = get_random_u32(); /* * FIXME diff --git a/fs/ocfs2/xattr.c b/fs/ocfs2/xattr.c index 4d55ad963ac5..86cfd4c2adf9 100644 --- a/fs/ocfs2/xattr.c +++ b/fs/ocfs2/xattr.c @@ -911,8 +911,8 @@ static int ocfs2_xattr_list_entry(struct super_block *sb, total_len = prefix_len + name_len + 1; *result += total_len; - /* we are just looking for how big our buffer needs to be */ - if (!size) + /* No buffer means we are only looking for the required size. */ + if (!buffer) return 0; if (*result > size) diff --git a/fs/proc/array.c b/fs/proc/array.c index f447e734612a..90fb0c6b5f99 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -280,7 +280,7 @@ static inline void task_sig(struct seq_file *m, struct task_struct *p) blocked = p->blocked; collect_sigign_sigcatch(p, &ignored, &caught); num_threads = get_nr_threads(p); - rcu_read_lock(); /* FIXME: is this correct? */ + rcu_read_lock(); qsize = get_rlimit_value(task_ucounts(p), UCOUNT_RLIMIT_SIGPENDING); rcu_read_unlock(); qlim = task_rlimit(p, RLIMIT_SIGPENDING); diff --git a/fs/ubifs/gc.c b/fs/ubifs/gc.c index 0bf08b7755b8..933c79b5cd6b 100644 --- a/fs/ubifs/gc.c +++ b/fs/ubifs/gc.c @@ -109,7 +109,6 @@ static int data_nodes_cmp(void *priv, const struct list_head *a, struct ubifs_info *c = priv; struct ubifs_scan_node *sa, *sb; - cond_resched(); if (a == b) return 0; @@ -153,7 +152,6 @@ static int nondata_nodes_cmp(void *priv, const struct list_head *a, struct ubifs_info *c = priv; struct ubifs_scan_node *sa, *sb; - cond_resched(); if (a == b) return 0; diff --git a/fs/ubifs/replay.c b/fs/ubifs/replay.c index a9a568f4a868..263045e05cf1 100644 --- a/fs/ubifs/replay.c +++ b/fs/ubifs/replay.c @@ -305,7 +305,6 @@ static int replay_entries_cmp(void *priv, const struct list_head *a, struct ubifs_info *c = priv; struct replay_entry *ra, *rb; - cond_resched(); if (a == b) return 0; diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild index 9aff61e7b8f2..2c53a1e0b760 100644 --- a/include/asm-generic/Kbuild +++ b/include/asm-generic/Kbuild @@ -65,4 +65,3 @@ mandatory-y += vermagic.h mandatory-y += vga.h mandatory-y += video.h mandatory-y += word-at-a-time.h -mandatory-y += xor.h diff --git a/include/asm-generic/xor.h b/include/asm-generic/xor.h deleted file mode 100644 index 44509d48fca2..000000000000 --- a/include/asm-generic/xor.h +++ /dev/null @@ -1,738 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * include/asm-generic/xor.h - * - * Generic optimized RAID-5 checksumming functions. - */ - -#include - -static void -xor_8regs_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2) -{ - long lines = bytes / (sizeof (long)) / 8; - - do { - p1[0] ^= p2[0]; - p1[1] ^= p2[1]; - p1[2] ^= p2[2]; - p1[3] ^= p2[3]; - p1[4] ^= p2[4]; - p1[5] ^= p2[5]; - p1[6] ^= p2[6]; - p1[7] ^= p2[7]; - p1 += 8; - p2 += 8; - } while (--lines > 0); -} - -static void -xor_8regs_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3) -{ - long lines = bytes / (sizeof (long)) / 8; - - do { - p1[0] ^= p2[0] ^ p3[0]; - p1[1] ^= p2[1] ^ p3[1]; - p1[2] ^= p2[2] ^ p3[2]; - p1[3] ^= p2[3] ^ p3[3]; - p1[4] ^= p2[4] ^ p3[4]; - p1[5] ^= p2[5] ^ p3[5]; - p1[6] ^= p2[6] ^ p3[6]; - p1[7] ^= p2[7] ^ p3[7]; - p1 += 8; - p2 += 8; - p3 += 8; - } while (--lines > 0); -} - -static void -xor_8regs_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4) -{ - long lines = bytes / (sizeof (long)) / 8; - - do { - p1[0] ^= p2[0] ^ p3[0] ^ p4[0]; - p1[1] ^= p2[1] ^ p3[1] ^ p4[1]; - p1[2] ^= p2[2] ^ p3[2] ^ p4[2]; - p1[3] ^= p2[3] ^ p3[3] ^ p4[3]; - p1[4] ^= p2[4] ^ p3[4] ^ p4[4]; - p1[5] ^= p2[5] ^ p3[5] ^ p4[5]; - p1[6] ^= p2[6] ^ p3[6] ^ p4[6]; - p1[7] ^= p2[7] ^ p3[7] ^ p4[7]; - p1 += 8; - p2 += 8; - p3 += 8; - p4 += 8; - } while (--lines > 0); -} - -static void -xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5) -{ - long lines = bytes / (sizeof (long)) / 8; - - do { - p1[0] ^= p2[0] ^ p3[0] ^ p4[0] ^ p5[0]; - p1[1] ^= p2[1] ^ p3[1] ^ p4[1] ^ p5[1]; - p1[2] ^= p2[2] ^ p3[2] ^ p4[2] ^ p5[2]; - p1[3] ^= p2[3] ^ p3[3] ^ p4[3] ^ p5[3]; - p1[4] ^= p2[4] ^ p3[4] ^ p4[4] ^ p5[4]; - p1[5] ^= p2[5] ^ p3[5] ^ p4[5] ^ p5[5]; - p1[6] ^= p2[6] ^ p3[6] ^ p4[6] ^ p5[6]; - p1[7] ^= p2[7] ^ p3[7] ^ p4[7] ^ p5[7]; - p1 += 8; - p2 += 8; - p3 += 8; - p4 += 8; - p5 += 8; - } while (--lines > 0); -} - -static void -xor_32regs_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2) -{ - long lines = bytes / (sizeof (long)) / 8; - - do { - register long d0, d1, d2, d3, d4, d5, d6, d7; - d0 = p1[0]; /* Pull the stuff into registers */ - d1 = p1[1]; /* ... in bursts, if possible. */ - d2 = p1[2]; - d3 = p1[3]; - d4 = p1[4]; - d5 = p1[5]; - d6 = p1[6]; - d7 = p1[7]; - d0 ^= p2[0]; - d1 ^= p2[1]; - d2 ^= p2[2]; - d3 ^= p2[3]; - d4 ^= p2[4]; - d5 ^= p2[5]; - d6 ^= p2[6]; - d7 ^= p2[7]; - p1[0] = d0; /* Store the result (in bursts) */ - p1[1] = d1; - p1[2] = d2; - p1[3] = d3; - p1[4] = d4; - p1[5] = d5; - p1[6] = d6; - p1[7] = d7; - p1 += 8; - p2 += 8; - } while (--lines > 0); -} - -static void -xor_32regs_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3) -{ - long lines = bytes / (sizeof (long)) / 8; - - do { - register long d0, d1, d2, d3, d4, d5, d6, d7; - d0 = p1[0]; /* Pull the stuff into registers */ - d1 = p1[1]; /* ... in bursts, if possible. */ - d2 = p1[2]; - d3 = p1[3]; - d4 = p1[4]; - d5 = p1[5]; - d6 = p1[6]; - d7 = p1[7]; - d0 ^= p2[0]; - d1 ^= p2[1]; - d2 ^= p2[2]; - d3 ^= p2[3]; - d4 ^= p2[4]; - d5 ^= p2[5]; - d6 ^= p2[6]; - d7 ^= p2[7]; - d0 ^= p3[0]; - d1 ^= p3[1]; - d2 ^= p3[2]; - d3 ^= p3[3]; - d4 ^= p3[4]; - d5 ^= p3[5]; - d6 ^= p3[6]; - d7 ^= p3[7]; - p1[0] = d0; /* Store the result (in bursts) */ - p1[1] = d1; - p1[2] = d2; - p1[3] = d3; - p1[4] = d4; - p1[5] = d5; - p1[6] = d6; - p1[7] = d7; - p1 += 8; - p2 += 8; - p3 += 8; - } while (--lines > 0); -} - -static void -xor_32regs_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4) -{ - long lines = bytes / (sizeof (long)) / 8; - - do { - register long d0, d1, d2, d3, d4, d5, d6, d7; - d0 = p1[0]; /* Pull the stuff into registers */ - d1 = p1[1]; /* ... in bursts, if possible. */ - d2 = p1[2]; - d3 = p1[3]; - d4 = p1[4]; - d5 = p1[5]; - d6 = p1[6]; - d7 = p1[7]; - d0 ^= p2[0]; - d1 ^= p2[1]; - d2 ^= p2[2]; - d3 ^= p2[3]; - d4 ^= p2[4]; - d5 ^= p2[5]; - d6 ^= p2[6]; - d7 ^= p2[7]; - d0 ^= p3[0]; - d1 ^= p3[1]; - d2 ^= p3[2]; - d3 ^= p3[3]; - d4 ^= p3[4]; - d5 ^= p3[5]; - d6 ^= p3[6]; - d7 ^= p3[7]; - d0 ^= p4[0]; - d1 ^= p4[1]; - d2 ^= p4[2]; - d3 ^= p4[3]; - d4 ^= p4[4]; - d5 ^= p4[5]; - d6 ^= p4[6]; - d7 ^= p4[7]; - p1[0] = d0; /* Store the result (in bursts) */ - p1[1] = d1; - p1[2] = d2; - p1[3] = d3; - p1[4] = d4; - p1[5] = d5; - p1[6] = d6; - p1[7] = d7; - p1 += 8; - p2 += 8; - p3 += 8; - p4 += 8; - } while (--lines > 0); -} - -static void -xor_32regs_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5) -{ - long lines = bytes / (sizeof (long)) / 8; - - do { - register long d0, d1, d2, d3, d4, d5, d6, d7; - d0 = p1[0]; /* Pull the stuff into registers */ - d1 = p1[1]; /* ... in bursts, if possible. */ - d2 = p1[2]; - d3 = p1[3]; - d4 = p1[4]; - d5 = p1[5]; - d6 = p1[6]; - d7 = p1[7]; - d0 ^= p2[0]; - d1 ^= p2[1]; - d2 ^= p2[2]; - d3 ^= p2[3]; - d4 ^= p2[4]; - d5 ^= p2[5]; - d6 ^= p2[6]; - d7 ^= p2[7]; - d0 ^= p3[0]; - d1 ^= p3[1]; - d2 ^= p3[2]; - d3 ^= p3[3]; - d4 ^= p3[4]; - d5 ^= p3[5]; - d6 ^= p3[6]; - d7 ^= p3[7]; - d0 ^= p4[0]; - d1 ^= p4[1]; - d2 ^= p4[2]; - d3 ^= p4[3]; - d4 ^= p4[4]; - d5 ^= p4[5]; - d6 ^= p4[6]; - d7 ^= p4[7]; - d0 ^= p5[0]; - d1 ^= p5[1]; - d2 ^= p5[2]; - d3 ^= p5[3]; - d4 ^= p5[4]; - d5 ^= p5[5]; - d6 ^= p5[6]; - d7 ^= p5[7]; - p1[0] = d0; /* Store the result (in bursts) */ - p1[1] = d1; - p1[2] = d2; - p1[3] = d3; - p1[4] = d4; - p1[5] = d5; - p1[6] = d6; - p1[7] = d7; - p1 += 8; - p2 += 8; - p3 += 8; - p4 += 8; - p5 += 8; - } while (--lines > 0); -} - -static void -xor_8regs_p_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2) -{ - long lines = bytes / (sizeof (long)) / 8 - 1; - prefetchw(p1); - prefetch(p2); - - do { - prefetchw(p1+8); - prefetch(p2+8); - once_more: - p1[0] ^= p2[0]; - p1[1] ^= p2[1]; - p1[2] ^= p2[2]; - p1[3] ^= p2[3]; - p1[4] ^= p2[4]; - p1[5] ^= p2[5]; - p1[6] ^= p2[6]; - p1[7] ^= p2[7]; - p1 += 8; - p2 += 8; - } while (--lines > 0); - if (lines == 0) - goto once_more; -} - -static void -xor_8regs_p_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3) -{ - long lines = bytes / (sizeof (long)) / 8 - 1; - prefetchw(p1); - prefetch(p2); - prefetch(p3); - - do { - prefetchw(p1+8); - prefetch(p2+8); - prefetch(p3+8); - once_more: - p1[0] ^= p2[0] ^ p3[0]; - p1[1] ^= p2[1] ^ p3[1]; - p1[2] ^= p2[2] ^ p3[2]; - p1[3] ^= p2[3] ^ p3[3]; - p1[4] ^= p2[4] ^ p3[4]; - p1[5] ^= p2[5] ^ p3[5]; - p1[6] ^= p2[6] ^ p3[6]; - p1[7] ^= p2[7] ^ p3[7]; - p1 += 8; - p2 += 8; - p3 += 8; - } while (--lines > 0); - if (lines == 0) - goto once_more; -} - -static void -xor_8regs_p_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4) -{ - long lines = bytes / (sizeof (long)) / 8 - 1; - - prefetchw(p1); - prefetch(p2); - prefetch(p3); - prefetch(p4); - - do { - prefetchw(p1+8); - prefetch(p2+8); - prefetch(p3+8); - prefetch(p4+8); - once_more: - p1[0] ^= p2[0] ^ p3[0] ^ p4[0]; - p1[1] ^= p2[1] ^ p3[1] ^ p4[1]; - p1[2] ^= p2[2] ^ p3[2] ^ p4[2]; - p1[3] ^= p2[3] ^ p3[3] ^ p4[3]; - p1[4] ^= p2[4] ^ p3[4] ^ p4[4]; - p1[5] ^= p2[5] ^ p3[5] ^ p4[5]; - p1[6] ^= p2[6] ^ p3[6] ^ p4[6]; - p1[7] ^= p2[7] ^ p3[7] ^ p4[7]; - p1 += 8; - p2 += 8; - p3 += 8; - p4 += 8; - } while (--lines > 0); - if (lines == 0) - goto once_more; -} - -static void -xor_8regs_p_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5) -{ - long lines = bytes / (sizeof (long)) / 8 - 1; - - prefetchw(p1); - prefetch(p2); - prefetch(p3); - prefetch(p4); - prefetch(p5); - - do { - prefetchw(p1+8); - prefetch(p2+8); - prefetch(p3+8); - prefetch(p4+8); - prefetch(p5+8); - once_more: - p1[0] ^= p2[0] ^ p3[0] ^ p4[0] ^ p5[0]; - p1[1] ^= p2[1] ^ p3[1] ^ p4[1] ^ p5[1]; - p1[2] ^= p2[2] ^ p3[2] ^ p4[2] ^ p5[2]; - p1[3] ^= p2[3] ^ p3[3] ^ p4[3] ^ p5[3]; - p1[4] ^= p2[4] ^ p3[4] ^ p4[4] ^ p5[4]; - p1[5] ^= p2[5] ^ p3[5] ^ p4[5] ^ p5[5]; - p1[6] ^= p2[6] ^ p3[6] ^ p4[6] ^ p5[6]; - p1[7] ^= p2[7] ^ p3[7] ^ p4[7] ^ p5[7]; - p1 += 8; - p2 += 8; - p3 += 8; - p4 += 8; - p5 += 8; - } while (--lines > 0); - if (lines == 0) - goto once_more; -} - -static void -xor_32regs_p_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2) -{ - long lines = bytes / (sizeof (long)) / 8 - 1; - - prefetchw(p1); - prefetch(p2); - - do { - register long d0, d1, d2, d3, d4, d5, d6, d7; - - prefetchw(p1+8); - prefetch(p2+8); - once_more: - d0 = p1[0]; /* Pull the stuff into registers */ - d1 = p1[1]; /* ... in bursts, if possible. */ - d2 = p1[2]; - d3 = p1[3]; - d4 = p1[4]; - d5 = p1[5]; - d6 = p1[6]; - d7 = p1[7]; - d0 ^= p2[0]; - d1 ^= p2[1]; - d2 ^= p2[2]; - d3 ^= p2[3]; - d4 ^= p2[4]; - d5 ^= p2[5]; - d6 ^= p2[6]; - d7 ^= p2[7]; - p1[0] = d0; /* Store the result (in bursts) */ - p1[1] = d1; - p1[2] = d2; - p1[3] = d3; - p1[4] = d4; - p1[5] = d5; - p1[6] = d6; - p1[7] = d7; - p1 += 8; - p2 += 8; - } while (--lines > 0); - if (lines == 0) - goto once_more; -} - -static void -xor_32regs_p_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3) -{ - long lines = bytes / (sizeof (long)) / 8 - 1; - - prefetchw(p1); - prefetch(p2); - prefetch(p3); - - do { - register long d0, d1, d2, d3, d4, d5, d6, d7; - - prefetchw(p1+8); - prefetch(p2+8); - prefetch(p3+8); - once_more: - d0 = p1[0]; /* Pull the stuff into registers */ - d1 = p1[1]; /* ... in bursts, if possible. */ - d2 = p1[2]; - d3 = p1[3]; - d4 = p1[4]; - d5 = p1[5]; - d6 = p1[6]; - d7 = p1[7]; - d0 ^= p2[0]; - d1 ^= p2[1]; - d2 ^= p2[2]; - d3 ^= p2[3]; - d4 ^= p2[4]; - d5 ^= p2[5]; - d6 ^= p2[6]; - d7 ^= p2[7]; - d0 ^= p3[0]; - d1 ^= p3[1]; - d2 ^= p3[2]; - d3 ^= p3[3]; - d4 ^= p3[4]; - d5 ^= p3[5]; - d6 ^= p3[6]; - d7 ^= p3[7]; - p1[0] = d0; /* Store the result (in bursts) */ - p1[1] = d1; - p1[2] = d2; - p1[3] = d3; - p1[4] = d4; - p1[5] = d5; - p1[6] = d6; - p1[7] = d7; - p1 += 8; - p2 += 8; - p3 += 8; - } while (--lines > 0); - if (lines == 0) - goto once_more; -} - -static void -xor_32regs_p_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4) -{ - long lines = bytes / (sizeof (long)) / 8 - 1; - - prefetchw(p1); - prefetch(p2); - prefetch(p3); - prefetch(p4); - - do { - register long d0, d1, d2, d3, d4, d5, d6, d7; - - prefetchw(p1+8); - prefetch(p2+8); - prefetch(p3+8); - prefetch(p4+8); - once_more: - d0 = p1[0]; /* Pull the stuff into registers */ - d1 = p1[1]; /* ... in bursts, if possible. */ - d2 = p1[2]; - d3 = p1[3]; - d4 = p1[4]; - d5 = p1[5]; - d6 = p1[6]; - d7 = p1[7]; - d0 ^= p2[0]; - d1 ^= p2[1]; - d2 ^= p2[2]; - d3 ^= p2[3]; - d4 ^= p2[4]; - d5 ^= p2[5]; - d6 ^= p2[6]; - d7 ^= p2[7]; - d0 ^= p3[0]; - d1 ^= p3[1]; - d2 ^= p3[2]; - d3 ^= p3[3]; - d4 ^= p3[4]; - d5 ^= p3[5]; - d6 ^= p3[6]; - d7 ^= p3[7]; - d0 ^= p4[0]; - d1 ^= p4[1]; - d2 ^= p4[2]; - d3 ^= p4[3]; - d4 ^= p4[4]; - d5 ^= p4[5]; - d6 ^= p4[6]; - d7 ^= p4[7]; - p1[0] = d0; /* Store the result (in bursts) */ - p1[1] = d1; - p1[2] = d2; - p1[3] = d3; - p1[4] = d4; - p1[5] = d5; - p1[6] = d6; - p1[7] = d7; - p1 += 8; - p2 += 8; - p3 += 8; - p4 += 8; - } while (--lines > 0); - if (lines == 0) - goto once_more; -} - -static void -xor_32regs_p_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5) -{ - long lines = bytes / (sizeof (long)) / 8 - 1; - - prefetchw(p1); - prefetch(p2); - prefetch(p3); - prefetch(p4); - prefetch(p5); - - do { - register long d0, d1, d2, d3, d4, d5, d6, d7; - - prefetchw(p1+8); - prefetch(p2+8); - prefetch(p3+8); - prefetch(p4+8); - prefetch(p5+8); - once_more: - d0 = p1[0]; /* Pull the stuff into registers */ - d1 = p1[1]; /* ... in bursts, if possible. */ - d2 = p1[2]; - d3 = p1[3]; - d4 = p1[4]; - d5 = p1[5]; - d6 = p1[6]; - d7 = p1[7]; - d0 ^= p2[0]; - d1 ^= p2[1]; - d2 ^= p2[2]; - d3 ^= p2[3]; - d4 ^= p2[4]; - d5 ^= p2[5]; - d6 ^= p2[6]; - d7 ^= p2[7]; - d0 ^= p3[0]; - d1 ^= p3[1]; - d2 ^= p3[2]; - d3 ^= p3[3]; - d4 ^= p3[4]; - d5 ^= p3[5]; - d6 ^= p3[6]; - d7 ^= p3[7]; - d0 ^= p4[0]; - d1 ^= p4[1]; - d2 ^= p4[2]; - d3 ^= p4[3]; - d4 ^= p4[4]; - d5 ^= p4[5]; - d6 ^= p4[6]; - d7 ^= p4[7]; - d0 ^= p5[0]; - d1 ^= p5[1]; - d2 ^= p5[2]; - d3 ^= p5[3]; - d4 ^= p5[4]; - d5 ^= p5[5]; - d6 ^= p5[6]; - d7 ^= p5[7]; - p1[0] = d0; /* Store the result (in bursts) */ - p1[1] = d1; - p1[2] = d2; - p1[3] = d3; - p1[4] = d4; - p1[5] = d5; - p1[6] = d6; - p1[7] = d7; - p1 += 8; - p2 += 8; - p3 += 8; - p4 += 8; - p5 += 8; - } while (--lines > 0); - if (lines == 0) - goto once_more; -} - -static struct xor_block_template xor_block_8regs = { - .name = "8regs", - .do_2 = xor_8regs_2, - .do_3 = xor_8regs_3, - .do_4 = xor_8regs_4, - .do_5 = xor_8regs_5, -}; - -static struct xor_block_template xor_block_32regs = { - .name = "32regs", - .do_2 = xor_32regs_2, - .do_3 = xor_32regs_3, - .do_4 = xor_32regs_4, - .do_5 = xor_32regs_5, -}; - -static struct xor_block_template xor_block_8regs_p __maybe_unused = { - .name = "8regs_prefetch", - .do_2 = xor_8regs_p_2, - .do_3 = xor_8regs_p_3, - .do_4 = xor_8regs_p_4, - .do_5 = xor_8regs_p_5, -}; - -static struct xor_block_template xor_block_32regs_p __maybe_unused = { - .name = "32regs_prefetch", - .do_2 = xor_32regs_p_2, - .do_3 = xor_32regs_p_3, - .do_4 = xor_32regs_p_4, - .do_5 = xor_32regs_p_5, -}; - -#define XOR_TRY_TEMPLATES \ - do { \ - xor_speed(&xor_block_8regs); \ - xor_speed(&xor_block_8regs_p); \ - xor_speed(&xor_block_32regs); \ - xor_speed(&xor_block_32regs_p); \ - } while (0) diff --git a/include/linux/crash_core.h b/include/linux/crash_core.h index d35726d6a415..c1dee3f971a9 100644 --- a/include/linux/crash_core.h +++ b/include/linux/crash_core.h @@ -34,13 +34,6 @@ static inline void arch_kexec_protect_crashkres(void) { } static inline void arch_kexec_unprotect_crashkres(void) { } #endif -#ifdef CONFIG_CRASH_DM_CRYPT -int crash_load_dm_crypt_keys(struct kimage *image); -ssize_t dm_crypt_keys_read(char *buf, size_t count, u64 *ppos); -#else -static inline int crash_load_dm_crypt_keys(struct kimage *image) {return 0; } -#endif - #ifndef arch_crash_handle_hotplug_event static inline void arch_crash_handle_hotplug_event(struct kimage *image, void *arg) { } #endif @@ -96,4 +89,11 @@ static inline void crash_save_cpu(struct pt_regs *regs, int cpu) {}; static inline int kimage_crash_copy_vmcoreinfo(struct kimage *image) { return 0; }; #endif /* CONFIG_CRASH_DUMP*/ +#ifdef CONFIG_CRASH_DM_CRYPT +int crash_load_dm_crypt_keys(struct kimage *image); +ssize_t dm_crypt_keys_read(char *buf, size_t count, u64 *ppos); +#else +static inline int crash_load_dm_crypt_keys(struct kimage *image) {return 0; } +#endif + #endif /* LINUX_CRASH_CORE_H */ diff --git a/include/linux/nmi.h b/include/linux/nmi.h index 207156f2143c..bc1162895f35 100644 --- a/include/linux/nmi.h +++ b/include/linux/nmi.h @@ -21,6 +21,7 @@ void lockup_detector_soft_poweroff(void); extern int watchdog_user_enabled; extern int watchdog_thresh; extern unsigned long watchdog_enabled; +extern int watchdog_hardlockup_miss_thresh; extern struct cpumask watchdog_cpumask; extern unsigned long *watchdog_cpumask_bits; diff --git a/include/linux/raid/xor.h b/include/linux/raid/xor.h index 51b811b62322..870558c9d36e 100644 --- a/include/linux/raid/xor.h +++ b/include/linux/raid/xor.h @@ -2,29 +2,6 @@ #ifndef _XOR_H #define _XOR_H -#define MAX_XOR_BLOCKS 4 +void xor_gen(void *dest, void **srcs, unsigned int src_cnt, unsigned int bytes); -extern void xor_blocks(unsigned int count, unsigned int bytes, - void *dest, void **srcs); - -struct xor_block_template { - struct xor_block_template *next; - const char *name; - int speed; - void (*do_2)(unsigned long, unsigned long * __restrict, - const unsigned long * __restrict); - void (*do_3)(unsigned long, unsigned long * __restrict, - const unsigned long * __restrict, - const unsigned long * __restrict); - void (*do_4)(unsigned long, unsigned long * __restrict, - const unsigned long * __restrict, - const unsigned long * __restrict, - const unsigned long * __restrict); - void (*do_5)(unsigned long, unsigned long * __restrict, - const unsigned long * __restrict, - const unsigned long * __restrict, - const unsigned long * __restrict, - const unsigned long * __restrict); -}; - -#endif +#endif /* _XOR_H */ diff --git a/kernel/crash_core.c b/kernel/crash_core.c index 2c1a3791e410..4f21fc3b108b 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -27,8 +27,6 @@ #include #include -#include - #include "kallsyms_internal.h" #include "kexec_internal.h" diff --git a/kernel/crash_dump_dm_crypt.c b/kernel/crash_dump_dm_crypt.c index a20d4097744a..cb875ddb6ba6 100644 --- a/kernel/crash_dump_dm_crypt.c +++ b/kernel/crash_dump_dm_crypt.c @@ -6,6 +6,7 @@ #include #include #include +#include #define KEY_NUM_MAX 128 /* maximum dm crypt keys */ #define KEY_SIZE_MAX 256 /* maximum dm crypt key size */ @@ -115,7 +116,7 @@ static int restore_dm_crypt_keys_to_thread_keyring(void) addr = dm_crypt_keys_addr; dm_crypt_keys_read((char *)&key_count, sizeof(key_count), &addr); - if (key_count < 0 || key_count > KEY_NUM_MAX) { + if (key_count > KEY_NUM_MAX) { kexec_dprintk("Failed to read the number of dm-crypt keys\n"); return -1; } @@ -139,7 +140,7 @@ static int restore_dm_crypt_keys_to_thread_keyring(void) return 0; } -static int read_key_from_user_keying(struct dm_crypt_key *dm_key) +static int read_key_from_user_keyring(struct dm_crypt_key *dm_key) { const struct user_key_payload *ukp; struct key *key; @@ -189,7 +190,7 @@ static inline struct config_key *to_config_key(struct config_item *item) static ssize_t config_key_description_show(struct config_item *item, char *page) { - return sprintf(page, "%s\n", to_config_key(item)->description); + return sysfs_emit(page, "%s\n", to_config_key(item)->description); } static ssize_t config_key_description_store(struct config_item *item, @@ -265,7 +266,7 @@ static struct config_item *config_keys_make_item(struct config_group *group, static ssize_t config_keys_count_show(struct config_item *item, char *page) { - return sprintf(page, "%d\n", key_count); + return sysfs_emit(page, "%d\n", key_count); } CONFIGFS_ATTR_RO(config_keys_, count); @@ -274,7 +275,7 @@ static bool is_dm_key_reused; static ssize_t config_keys_reuse_show(struct config_item *item, char *page) { - return sprintf(page, "%d\n", is_dm_key_reused); + return sysfs_emit(page, "%d\n", is_dm_key_reused); } static ssize_t config_keys_reuse_store(struct config_item *item, @@ -321,7 +322,7 @@ static bool restore; static ssize_t config_keys_restore_show(struct config_item *item, char *page) { - return sprintf(page, "%d\n", restore); + return sysfs_emit(page, "%d\n", restore); } static ssize_t config_keys_restore_store(struct config_item *item, @@ -387,7 +388,7 @@ static int build_keys_header(void) strscpy(keys_header->keys[i].key_desc, key->description, KEY_DESC_MAX_LEN); - r = read_key_from_user_keying(&keys_header->keys[i]); + r = read_key_from_user_keyring(&keys_header->keys[i]); if (r != 0) { kexec_dprintk("Failed to read key %s\n", keys_header->keys[i].key_desc); @@ -414,14 +415,16 @@ int crash_load_dm_crypt_keys(struct kimage *image) if (key_count <= 0) { kexec_dprintk("No dm-crypt keys\n"); - return -ENOENT; + return 0; } if (!is_dm_key_reused) { image->dm_crypt_keys_addr = 0; r = build_keys_header(); - if (r) + if (r) { + pr_err("Failed to build dm-crypt keys header, ret=%d\n", r); return r; + } } kbuf.buffer = keys_header; @@ -432,6 +435,7 @@ int crash_load_dm_crypt_keys(struct kimage *image) kbuf.mem = KEXEC_BUF_MEM_UNKNOWN; r = kexec_add_buffer(&kbuf); if (r) { + pr_err("Failed to call kexec_add_buffer, ret=%d\n", r); kvfree((void *)kbuf.buffer); return r; } diff --git a/kernel/crash_reserve.c b/kernel/crash_reserve.c index 62e60e0223cf..eee37a11380c 100644 --- a/kernel/crash_reserve.c +++ b/kernel/crash_reserve.c @@ -20,8 +20,6 @@ #include #include -#include - #include "kallsyms_internal.h" #include "kexec_internal.h" diff --git a/kernel/exit.c b/kernel/exit.c index 31b714c3a791..25e9cb6de7e7 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -749,14 +749,12 @@ static void exit_notify(struct task_struct *tsk, int group_dead) tsk->exit_state = EXIT_ZOMBIE; if (unlikely(tsk->ptrace)) { - int sig = thread_group_leader(tsk) && - thread_group_empty(tsk) && - !ptrace_reparented(tsk) ? - tsk->exit_signal : SIGCHLD; + int sig = thread_group_empty(tsk) && !ptrace_reparented(tsk) + ? tsk->exit_signal : SIGCHLD; autoreap = do_notify_parent(tsk, sig); } else if (thread_group_leader(tsk)) { autoreap = thread_group_empty(tsk) && - do_notify_parent(tsk, tsk->exit_signal); + do_notify_parent(tsk, tsk->exit_signal); } else { autoreap = true; /* untraced sub-thread */ diff --git a/kernel/fork.c b/kernel/fork.c index fe3821160f9a..f1ad69c6dc2d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -347,7 +347,7 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) stack = kasan_reset_tag(vm_area->addr); /* Clear stale pointers from reused stack. */ - memset(stack, 0, THREAD_SIZE); + clear_pages(vm_area->addr, vm_area->nr_pages); tsk->stack_vm_area = vm_area; tsk->stack = stack; @@ -1016,13 +1016,14 @@ free_tsk: __cacheline_aligned_in_smp DEFINE_SPINLOCK(mmlist_lock); -static unsigned long default_dump_filter = MMF_DUMP_FILTER_DEFAULT; +static unsigned long coredump_filter = MMF_DUMP_FILTER_DEFAULT; static int __init coredump_filter_setup(char *s) { - default_dump_filter = - (simple_strtoul(s, NULL, 0) << MMF_DUMP_FILTER_SHIFT) & - MMF_DUMP_FILTER_MASK; + if (kstrtoul(s, 0, &coredump_filter)) + return 0; + coredump_filter <<= MMF_DUMP_FILTER_SHIFT; + coredump_filter &= MMF_DUMP_FILTER_MASK; return 1; } @@ -1108,7 +1109,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, __mm_flags_overwrite_word(mm, mmf_init_legacy_flags(flags)); mm->def_flags = current->mm->def_flags & VM_INIT_DEF_MASK; } else { - __mm_flags_overwrite_word(mm, default_dump_filter); + __mm_flags_overwrite_word(mm, coredump_filter); mm->def_flags = 0; } @@ -2435,7 +2436,11 @@ __latent_entropy struct task_struct *copy_process( rseq_fork(p, clone_flags); - /* Don't start children in a dying pid namespace */ + /* + * If zap_pid_ns_processes() was called after alloc_pid(), the new + * child missed SIGKILL. If current is not in the same namespace, + * we can't rely on fatal_signal_pending() below. + */ if (unlikely(!(ns_of_pid(pid)->pid_allocated & PIDNS_ADDING))) { retval = -ENOMEM; goto bad_fork_core_free; @@ -3241,11 +3246,10 @@ int ksys_unshare(unsigned long unshare_flags) new_cred, new_fs); if (err) goto bad_unshare_cleanup_cred; - if (new_cred) { err = set_cred_ucounts(new_cred); if (err) - goto bad_unshare_cleanup_cred; + goto bad_unshare_cleanup_nsproxy; } if (new_fs || new_fd || do_sysvsem || new_cred || new_nsproxy) { @@ -3261,8 +3265,10 @@ int ksys_unshare(unsigned long unshare_flags) shm_init_task(current); } - if (new_nsproxy) + if (new_nsproxy) { switch_task_namespaces(current, new_nsproxy); + new_nsproxy = NULL; + } task_lock(current); @@ -3291,13 +3297,15 @@ int ksys_unshare(unsigned long unshare_flags) perf_event_namespaces(current); +bad_unshare_cleanup_nsproxy: + if (new_nsproxy) + put_nsproxy(new_nsproxy); bad_unshare_cleanup_cred: if (new_cred) put_cred(new_cred); bad_unshare_cleanup_fd: if (new_fd) put_files_struct(new_fd); - bad_unshare_cleanup_fs: if (new_fs) free_fs_struct(new_fs); diff --git a/kernel/hung_task.c b/kernel/hung_task.c index d2254c91450b..6fcc94ce4ca9 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -36,7 +36,7 @@ static int __read_mostly sysctl_hung_task_check_count = PID_MAX_LIMIT; /* * Total number of tasks detected as hung since boot: */ -static unsigned long __read_mostly sysctl_hung_task_detect_count; +static atomic_long_t sysctl_hung_task_detect_count = ATOMIC_LONG_INIT(0); /* * Limit number of tasks checked in a batch. @@ -223,37 +223,36 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti } #endif -static void check_hung_task(struct task_struct *t, unsigned long timeout, - unsigned long prev_detect_count) +/** + * hung_task_info - Print diagnostic details for a hung task + * @t: Pointer to the detected hung task. + * @timeout: Timeout threshold for detecting hung tasks + * @this_round_count: Count of hung tasks detected in the current iteration + * + * Print structured information about the specified hung task, if warnings + * are enabled or if the panic batch threshold is exceeded. + */ +static void hung_task_info(struct task_struct *t, unsigned long timeout, + unsigned long this_round_count) { - unsigned long total_hung_task; - - if (!task_is_hung(t, timeout)) - return; - - /* - * This counter tracks the total number of tasks detected as hung - * since boot. - */ - sysctl_hung_task_detect_count++; - - total_hung_task = sysctl_hung_task_detect_count - prev_detect_count; trace_sched_process_hang(t); - if (sysctl_hung_task_panic && total_hung_task >= sysctl_hung_task_panic) { + if (sysctl_hung_task_panic && this_round_count >= sysctl_hung_task_panic) { console_verbose(); hung_task_call_panic = true; } /* - * Ok, the task did not get scheduled for more than 2 minutes, - * complain: + * The given task did not get scheduled for more than + * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain + * accordingly */ if (sysctl_hung_task_warnings || hung_task_call_panic) { if (sysctl_hung_task_warnings > 0) sysctl_hung_task_warnings--; - pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n", - t->comm, t->pid, (jiffies - t->last_switch_time) / HZ); + pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", + t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", + (jiffies - t->last_switch_time) / HZ); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "), @@ -297,15 +296,14 @@ static bool rcu_lock_break(struct task_struct *g, struct task_struct *t) /* * Check whether a TASK_UNINTERRUPTIBLE does not get woken up for - * a really long time (120 seconds). If that happens, print out - * a warning. + * a really long time. If that happens, print out a warning. */ static void check_hung_uninterruptible_tasks(unsigned long timeout) { int max_count = sysctl_hung_task_check_count; unsigned long last_break = jiffies; struct task_struct *g, *t; - unsigned long prev_detect_count = sysctl_hung_task_detect_count; + unsigned long this_round_count; int need_warning = sysctl_hung_task_warnings; unsigned long si_mask = hung_task_si_mask; @@ -316,10 +314,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) if (test_taint(TAINT_DIE) || did_panic) return; - + this_round_count = 0; rcu_read_lock(); for_each_process_thread(g, t) { - if (!max_count--) goto unlock; if (time_after(jiffies, last_break + HUNG_TASK_LOCK_BREAK)) { @@ -328,12 +325,22 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) last_break = jiffies; } - check_hung_task(t, timeout, prev_detect_count); + if (task_is_hung(t, timeout)) { + /* + * Increment the global counter so that userspace could + * start migrating tasks ASAP. But count the current + * round separately because userspace could reset + * the global counter at any time. + */ + atomic_long_inc(&sysctl_hung_task_detect_count); + this_round_count++; + hung_task_info(t, timeout, this_round_count); + } } unlock: rcu_read_unlock(); - if (!(sysctl_hung_task_detect_count - prev_detect_count)) + if (!this_round_count) return; if (need_warning || hung_task_call_panic) { @@ -358,6 +365,46 @@ static long hung_timeout_jiffies(unsigned long last_checked, } #ifdef CONFIG_SYSCTL + +/** + * proc_dohung_task_detect_count - proc handler for hung_task_detect_count + * @table: Pointer to the struct ctl_table definition for this proc entry + * @dir: Flag indicating the operation + * @buffer: User space buffer for data transfer + * @lenp: Pointer to the length of the data being transferred + * @ppos: Pointer to the current file offset + * + * This handler is used for reading the current hung task detection count + * and for resetting it to zero when a write operation is performed using a + * zero value only. + * Return: 0 on success, or a negative error code on failure. + */ +static int proc_dohung_task_detect_count(const struct ctl_table *table, int dir, + void *buffer, size_t *lenp, loff_t *ppos) +{ + unsigned long detect_count; + struct ctl_table proxy_table; + int err; + + proxy_table = *table; + proxy_table.data = &detect_count; + + if (SYSCTL_KERN_TO_USER(dir)) + detect_count = atomic_long_read(&sysctl_hung_task_detect_count); + + err = proc_doulongvec_minmax(&proxy_table, dir, buffer, lenp, ppos); + if (err < 0) + return err; + + if (SYSCTL_USER_TO_KERN(dir)) { + if (detect_count) + return -EINVAL; + atomic_long_set(&sysctl_hung_task_detect_count, 0); + } + + return 0; +} + /* * Process updating of timeout sysctl */ @@ -438,10 +485,9 @@ static const struct ctl_table hung_task_sysctls[] = { }, { .procname = "hung_task_detect_count", - .data = &sysctl_hung_task_detect_count, .maxlen = sizeof(unsigned long), - .mode = 0444, - .proc_handler = proc_doulongvec_minmax, + .mode = 0644, + .proc_handler = proc_dohung_task_detect_count, }, { .procname = "hung_task_sys_info", diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index 2fea396d29b9..a43d2da0fe3e 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -47,7 +47,6 @@ #include #include -#include #include "kexec_internal.h" atomic_t __kexec_lock = ATOMIC_INIT(0); diff --git a/kernel/panic.c b/kernel/panic.c index c78600212b6c..20feada5319d 100644 --- a/kernel/panic.c +++ b/kernel/panic.c @@ -801,6 +801,8 @@ EXPORT_SYMBOL(panic); * Documentation/admin-guide/tainted-kernels.rst, including its * small shell script that prints the TAINT_FLAGS_COUNT bits of * /proc/sys/kernel/tainted. + * + * Also, update INIT_TAINT_BUF_MAX below. */ const struct taint_flag taint_flags[TAINT_FLAGS_COUNT] = { TAINT_FLAG(PROPRIETARY_MODULE, 'P', 'G'), @@ -854,15 +856,54 @@ static void print_tainted_seq(struct seq_buf *s, bool verbose) } } +/* The initial buffer can accommodate all taint flags in verbose + * mode, with some headroom. Once the allocator is available, the + * exact size is allocated dynamically; the initial buffer remains + * as a fallback if allocation fails. + * + * The verbose taint string currently requires up to 327 characters. + */ +#define INIT_TAINT_BUF_MAX 350 + +static char init_taint_buf[INIT_TAINT_BUF_MAX] __initdata; +static char *taint_buf __refdata = init_taint_buf; +static size_t taint_buf_size = INIT_TAINT_BUF_MAX; + +static __init int alloc_taint_buf(void) +{ + int i; + char *buf; + size_t size = 0; + + size += sizeof("Tainted: ") - 1; + for (i = 0; i < TAINT_FLAGS_COUNT; i++) { + size += 2; /* For ", " */ + size += 4; /* For "[%c]=" */ + size += strlen(taint_flags[i].desc); + } + + size += 1; /* For NULL terminator */ + + buf = kmalloc(size, GFP_KERNEL); + + if (!buf) { + panic("Failed to allocate taint string buffer"); + } + + taint_buf = buf; + taint_buf_size = size; + + return 0; +} +postcore_initcall(alloc_taint_buf); + static const char *_print_tainted(bool verbose) { - /* FIXME: what should the size be? */ - static char buf[sizeof(taint_flags)]; struct seq_buf s; BUILD_BUG_ON(ARRAY_SIZE(taint_flags) != TAINT_FLAGS_COUNT); - seq_buf_init(&s, buf, sizeof(buf)); + seq_buf_init(&s, taint_buf, taint_buf_size); print_tainted_seq(&s, verbose); diff --git a/kernel/pid.c b/kernel/pid.c index 677c84e319dd..fd5c2d4aa349 100644 --- a/kernel/pid.c +++ b/kernel/pid.c @@ -131,9 +131,8 @@ void free_pid(struct pid *pid) wake_up_process(READ_ONCE(ns->child_reaper)); break; case PIDNS_ADDING: - /* Handle a fork failure of the first process */ - WARN_ON(ns->child_reaper); - ns->pid_allocated = 0; + /* Only possible if the 1st fork fails */ + WARN_ON(READ_ONCE(ns->child_reaper)); break; } @@ -230,6 +229,10 @@ struct pid *alloc_pid(struct pid_namespace *ns, pid_t *arg_set_tid, retried_preload = false; idr_preload(GFP_KERNEL); spin_lock(&pidmap_lock); + /* For the case when the previous attempt to create init failed */ + if (ns->pid_allocated == PIDNS_ADDING) + idr_set_cursor(&ns->idr, 0); + for (tmp = ns, i = ns->level; i >= 0;) { int tid = set_tid[ns->level - i]; @@ -314,6 +317,11 @@ struct pid *alloc_pid(struct pid_namespace *ns, pid_t *arg_set_tid, * * This can't be done earlier because we need to preserve other * error conditions. + * + * We need this even if copy_process() does the same check. If two + * or more tasks from parent namespace try to inject a child into a + * dead namespace, one of free_pid() calls from the copy_process() + * error path may try to wakeup the possibly freed ns->child_reaper. */ retval = -ENOMEM; if (unlikely(!(ns->pid_allocated & PIDNS_ADDING))) @@ -341,10 +349,6 @@ out_free: idr_remove(&upid->ns->idr, upid->nr); } - /* On failure to allocate the first pid, reset the state */ - if (ns->pid_allocated == PIDNS_ADDING) - idr_set_cursor(&ns->idr, 0); - spin_unlock(&pidmap_lock); idr_preload_end(); diff --git a/kernel/signal.c b/kernel/signal.c index 21d881b95ffb..2d102e025883 100644 --- a/kernel/signal.c +++ b/kernel/signal.c @@ -1000,9 +1000,7 @@ static void complete_signal(int sig, struct task_struct *p, enum pid_type type) * Found a killable thread. If the signal will be fatal, * then start taking the whole group down immediately. */ - if (sig_fatal(p, sig) && - (signal->core_state || !(signal->flags & SIGNAL_GROUP_EXIT)) && - !sigismember(&t->real_blocked, sig) && + if (sig_fatal(p, sig) && !sigismember(&t->real_blocked, sig) && (sig == SIGKILL || !p->ptrace)) { /* * This signal will be fatal to the whole group. @@ -2173,13 +2171,13 @@ bool do_notify_parent(struct task_struct *tsk, int sig) bool autoreap = false; u64 utime, stime; - WARN_ON_ONCE(sig == -1); + if (WARN_ON_ONCE(!valid_signal(sig))) + return false; /* do_notify_parent_cldstop should have been called instead. */ WARN_ON_ONCE(task_is_stopped_or_traced(tsk)); - WARN_ON_ONCE(!tsk->ptrace && - (tsk->group_leader != tsk || !thread_group_empty(tsk))); + WARN_ON_ONCE(!tsk->ptrace && !thread_group_empty(tsk)); /* ptraced, or group-leader without sub-threads */ do_notify_pidfd(tsk); @@ -2259,7 +2257,7 @@ bool do_notify_parent(struct task_struct *tsk, int sig) * Send with __send_signal as si_pid and si_uid are in the * parent's namespaces. */ - if (valid_signal(sig) && sig) + if (sig) __send_signal_locked(sig, &info, tsk->parent, PIDTYPE_TGID, false); __wake_up_parent(tsk, tsk->parent); spin_unlock_irqrestore(&psig->siglock, flags); diff --git a/kernel/taskstats.c b/kernel/taskstats.c index 0cd680ccc7e5..73bd6a6a7893 100644 --- a/kernel/taskstats.c +++ b/kernel/taskstats.c @@ -649,6 +649,7 @@ void taskstats_exit(struct task_struct *tsk, int group_dead) goto err; memcpy(stats, tsk->signal->stats, sizeof(*stats)); + stats->version = TASKSTATS_VERSION; send: send_cpu_listeners(rep_skb, listeners); diff --git a/kernel/vmcore_info.c b/kernel/vmcore_info.c index 94e4ef75b1b2..8614430ca212 100644 --- a/kernel/vmcore_info.c +++ b/kernel/vmcore_info.c @@ -18,8 +18,6 @@ #include #include -#include - #include "kallsyms_internal.h" #include "kexec_internal.h" diff --git a/kernel/watchdog.c b/kernel/watchdog.c index 7d675781bc91..87dd5e0f6968 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -60,6 +60,13 @@ unsigned long *watchdog_cpumask_bits = cpumask_bits(&watchdog_cpumask); int __read_mostly sysctl_hardlockup_all_cpu_backtrace; # endif /* CONFIG_SMP */ +/* + * Number of consecutive missed interrupts before declaring a lockup. + * Default to 1 (immediate) for NMI/Perf. Buddy will overwrite this to 3. + */ +int __read_mostly watchdog_hardlockup_miss_thresh = 1; +EXPORT_SYMBOL_GPL(watchdog_hardlockup_miss_thresh); + /* * Should we panic when a soft-lockup or hard-lockup occurs: */ @@ -137,6 +144,7 @@ __setup("nmi_watchdog=", hardlockup_panic_setup); static DEFINE_PER_CPU(atomic_t, hrtimer_interrupts); static DEFINE_PER_CPU(int, hrtimer_interrupts_saved); +static DEFINE_PER_CPU(int, hrtimer_interrupts_missed); static DEFINE_PER_CPU(bool, watchdog_hardlockup_warned); static DEFINE_PER_CPU(bool, watchdog_hardlockup_touched); static unsigned long hard_lockup_nmi_warn; @@ -159,21 +167,33 @@ void watchdog_hardlockup_touch_cpu(unsigned int cpu) per_cpu(watchdog_hardlockup_touched, cpu) = true; } -static bool is_hardlockup(unsigned int cpu) +static void watchdog_hardlockup_update_reset(unsigned int cpu) { int hrint = atomic_read(&per_cpu(hrtimer_interrupts, cpu)); - if (per_cpu(hrtimer_interrupts_saved, cpu) == hrint) - return true; - /* * NOTE: we don't need any fancy atomic_t or READ_ONCE/WRITE_ONCE * for hrtimer_interrupts_saved. hrtimer_interrupts_saved is * written/read by a single CPU. */ per_cpu(hrtimer_interrupts_saved, cpu) = hrint; + per_cpu(hrtimer_interrupts_missed, cpu) = 0; +} - return false; +static bool is_hardlockup(unsigned int cpu) +{ + int hrint = atomic_read(&per_cpu(hrtimer_interrupts, cpu)); + + if (per_cpu(hrtimer_interrupts_saved, cpu) != hrint) { + watchdog_hardlockup_update_reset(cpu); + return false; + } + + per_cpu(hrtimer_interrupts_missed, cpu)++; + if (per_cpu(hrtimer_interrupts_missed, cpu) % watchdog_hardlockup_miss_thresh) + return false; + + return true; } static void watchdog_hardlockup_kick(void) @@ -187,8 +207,11 @@ static void watchdog_hardlockup_kick(void) void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) { int hardlockup_all_cpu_backtrace; + unsigned int this_cpu; + unsigned long flags; if (per_cpu(watchdog_hardlockup_touched, cpu)) { + watchdog_hardlockup_update_reset(cpu); per_cpu(watchdog_hardlockup_touched, cpu) = false; return; } @@ -201,74 +224,73 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) * fired multiple times before we overflow'd. If it hasn't * then this is a good indication the cpu is stuck */ - if (is_hardlockup(cpu)) { - unsigned int this_cpu = smp_processor_id(); - unsigned long flags; + if (!is_hardlockup(cpu)) { + per_cpu(watchdog_hardlockup_warned, cpu) = false; + return; + } #ifdef CONFIG_SYSFS - ++hardlockup_count; + ++hardlockup_count; #endif - /* - * A poorly behaving BPF scheduler can trigger hard lockup by - * e.g. putting numerous affinitized tasks in a single queue and - * directing all CPUs at it. The following call can return true - * only once when sched_ext is enabled and will immediately - * abort the BPF scheduler and print out a warning message. - */ - if (scx_hardlockup(cpu)) + /* + * A poorly behaving BPF scheduler can trigger hard lockup by + * e.g. putting numerous affinitized tasks in a single queue and + * directing all CPUs at it. The following call can return true + * only once when sched_ext is enabled and will immediately + * abort the BPF scheduler and print out a warning message. + */ + if (scx_hardlockup(cpu)) + return; + + /* Only print hardlockups once. */ + if (per_cpu(watchdog_hardlockup_warned, cpu)) + return; + + /* + * Prevent multiple hard-lockup reports if one cpu is already + * engaged in dumping all cpu back traces. + */ + if (hardlockup_all_cpu_backtrace) { + if (test_and_set_bit_lock(0, &hard_lockup_nmi_warn)) return; - - /* Only print hardlockups once. */ - if (per_cpu(watchdog_hardlockup_warned, cpu)) - return; - - /* - * Prevent multiple hard-lockup reports if one cpu is already - * engaged in dumping all cpu back traces. - */ - if (hardlockup_all_cpu_backtrace) { - if (test_and_set_bit_lock(0, &hard_lockup_nmi_warn)) - return; - } - - /* - * NOTE: we call printk_cpu_sync_get_irqsave() after printing - * the lockup message. While it would be nice to serialize - * that printout, we really want to make sure that if some - * other CPU somehow locked up while holding the lock associated - * with printk_cpu_sync_get_irqsave() that we can still at least - * get the message about the lockup out. - */ - pr_emerg("CPU%u: Watchdog detected hard LOCKUP on cpu %u\n", this_cpu, cpu); - printk_cpu_sync_get_irqsave(flags); - - print_modules(); - print_irqtrace_events(current); - if (cpu == this_cpu) { - if (regs) - show_regs(regs); - else - dump_stack(); - printk_cpu_sync_put_irqrestore(flags); - } else { - printk_cpu_sync_put_irqrestore(flags); - trigger_single_cpu_backtrace(cpu); - } - - if (hardlockup_all_cpu_backtrace) { - trigger_allbutcpu_cpu_backtrace(cpu); - if (!hardlockup_panic) - clear_bit_unlock(0, &hard_lockup_nmi_warn); - } - - sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT); - if (hardlockup_panic) - nmi_panic(regs, "Hard LOCKUP"); - - per_cpu(watchdog_hardlockup_warned, cpu) = true; - } else { - per_cpu(watchdog_hardlockup_warned, cpu) = false; } + + /* + * NOTE: we call printk_cpu_sync_get_irqsave() after printing + * the lockup message. While it would be nice to serialize + * that printout, we really want to make sure that if some + * other CPU somehow locked up while holding the lock associated + * with printk_cpu_sync_get_irqsave() that we can still at least + * get the message about the lockup out. + */ + this_cpu = smp_processor_id(); + pr_emerg("CPU%u: Watchdog detected hard LOCKUP on cpu %u\n", this_cpu, cpu); + printk_cpu_sync_get_irqsave(flags); + + print_modules(); + print_irqtrace_events(current); + if (cpu == this_cpu) { + if (regs) + show_regs(regs); + else + dump_stack(); + printk_cpu_sync_put_irqrestore(flags); + } else { + printk_cpu_sync_put_irqrestore(flags); + trigger_single_cpu_backtrace(cpu); + } + + if (hardlockup_all_cpu_backtrace) { + trigger_allbutcpu_cpu_backtrace(cpu); + if (!hardlockup_panic) + clear_bit_unlock(0, &hard_lockup_nmi_warn); + } + + sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT); + if (hardlockup_panic) + nmi_panic(regs, "Hard LOCKUP"); + + per_cpu(watchdog_hardlockup_warned, cpu) = true; } #else /* CONFIG_HARDLOCKUP_DETECTOR_COUNTS_HRTIMER */ diff --git a/kernel/watchdog_buddy.c b/kernel/watchdog_buddy.c index ee754d767c21..3a1e57080c1c 100644 --- a/kernel/watchdog_buddy.c +++ b/kernel/watchdog_buddy.c @@ -21,6 +21,7 @@ static unsigned int watchdog_next_cpu(unsigned int cpu) int __init watchdog_hardlockup_probe(void) { + watchdog_hardlockup_miss_thresh = 3; return 0; } @@ -86,14 +87,6 @@ void watchdog_buddy_check_hardlockup(int hrtimer_interrupts) { unsigned int next_cpu; - /* - * Test for hardlockups every 3 samples. The sample period is - * watchdog_thresh * 2 / 5, so 3 samples gets us back to slightly over - * watchdog_thresh (over by 20%). - */ - if (hrtimer_interrupts % 3 != 0) - return; - /* check for a hardlockup on the next CPU */ next_cpu = watchdog_next_cpu(smp_processor_id()); if (next_cpu >= nr_cpu_ids) diff --git a/lib/Kconfig b/lib/Kconfig index 0f2fb9610647..00a9509636c1 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -138,6 +138,7 @@ config TRACE_MMIO_ACCESS source "lib/crc/Kconfig" source "lib/crypto/Kconfig" +source "lib/raid/Kconfig" config XXHASH tristate @@ -625,9 +626,6 @@ config PLDMFW config ASN1_ENCODER tristate -config POLYNOMIAL - tristate - config FIRMWARE_TABLE bool diff --git a/lib/Makefile b/lib/Makefile index 72c90fca6fef..f33a24bf1c19 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -121,7 +121,7 @@ endif obj-$(CONFIG_DEBUG_INFO_REDUCED) += debug_info.o CFLAGS_debug_info.o += $(call cc-option, -femit-struct-debug-detailed=any) -obj-y += math/ crc/ crypto/ tests/ vdso/ +obj-y += math/ crc/ crypto/ tests/ vdso/ raid/ obj-$(CONFIG_GENERIC_IOMAP) += iomap.o obj-$(CONFIG_HAS_IOMEM) += iomap_copy.o devres.o @@ -244,8 +244,6 @@ obj-$(CONFIG_MEMREGION) += memregion.o obj-$(CONFIG_STMP_DEVICE) += stmp_device.o obj-$(CONFIG_IRQ_POLL) += irq_poll.o -obj-$(CONFIG_POLYNOMIAL) += polynomial.o - # stackdepot.c should not be instrumented or call instrumented functions. # Prevent the compiler from calling builtins like memcmp() or bcmp() from this # file. diff --git a/lib/bch.c b/lib/bch.c index 9561c0828802..c991c71c4cbd 100644 --- a/lib/bch.c +++ b/lib/bch.c @@ -392,7 +392,7 @@ static void compute_syndromes(struct bch_control *bch, uint32_t *ecc, for (j = 0; j < 2*t; j += 2) syn[j] ^= a_pow(bch, (j+1)*(i+s)); - poly ^= (1 << i); + poly ^= (1u << i); } } while (s > 0); @@ -612,7 +612,7 @@ static int find_poly_deg2_roots(struct bch_control *bch, struct gf_poly *poly, while (v) { i = deg(v); r ^= bch->xi_tab[i]; - v ^= (1 << i); + v ^= (1u << i); } /* verify root */ if ((gf_sqr(bch, r)^r) == u) { @@ -1116,7 +1116,7 @@ static void build_mod8_tables(struct bch_control *bch, const uint32_t *g) for (b = 0; b < 4; b++) { /* we want to compute (p(X).X^(8*b+deg(g))) mod g(X) */ tab = bch->mod8_tab + (b*256+i)*l; - data = i << (8*b); + data = (unsigned int)i << (8*b); while (data) { d = deg(data); /* subtract X^d.g(X) from p(X).X^(8*b+deg(g)) */ diff --git a/lib/bug.c b/lib/bug.c index aab9e6a40c5f..224f4cfa4aa3 100644 --- a/lib/bug.c +++ b/lib/bug.c @@ -251,7 +251,7 @@ static enum bug_trap_type __report_bug(struct bug_entry *bug, unsigned long buga if (file) pr_crit("kernel BUG at %s:%u!\n", file, line); else - pr_crit("Kernel BUG at %pB [verbose debug info unavailable]\n", + pr_crit("kernel BUG at %pB [verbose debug info unavailable]\n", (void *)bugaddr); return BUG_TRAP_TYPE_BUG; @@ -260,7 +260,7 @@ static enum bug_trap_type __report_bug(struct bug_entry *bug, unsigned long buga enum bug_trap_type report_bug_entry(struct bug_entry *bug, struct pt_regs *regs) { enum bug_trap_type ret; - bool rcu = false; + bool rcu; rcu = warn_rcu_enter(); ret = __report_bug(bug, bug_addr(bug), regs); @@ -272,7 +272,7 @@ enum bug_trap_type report_bug_entry(struct bug_entry *bug, struct pt_regs *regs) enum bug_trap_type report_bug(unsigned long bugaddr, struct pt_regs *regs) { enum bug_trap_type ret; - bool rcu = false; + bool rcu; rcu = warn_rcu_enter(); ret = __report_bug(NULL, bugaddr, regs); diff --git a/lib/decompress_bunzip2.c b/lib/decompress_bunzip2.c index ca736166f100..1288f146661f 100644 --- a/lib/decompress_bunzip2.c +++ b/lib/decompress_bunzip2.c @@ -135,7 +135,7 @@ static unsigned int INIT get_bits(struct bunzip_data *bd, char bits_wanted) } /* Avoid 32-bit overflow (dump bit buffer to top of output) */ if (bd->inbufBitCount >= 24) { - bits = bd->inbufBits&((1 << bd->inbufBitCount)-1); + bits = bd->inbufBits & ((1ULL << bd->inbufBitCount) - 1); bits_wanted -= bd->inbufBitCount; bits <<= bits_wanted; bd->inbufBitCount = 0; @@ -146,7 +146,7 @@ static unsigned int INIT get_bits(struct bunzip_data *bd, char bits_wanted) } /* Calculate result */ bd->inbufBitCount -= bits_wanted; - bits |= (bd->inbufBits >> bd->inbufBitCount)&((1 << bits_wanted)-1); + bits |= (bd->inbufBits >> bd->inbufBitCount) & ((1ULL << bits_wanted) - 1); return bits; } diff --git a/lib/glob.c b/lib/glob.c index aa57900d2062..7aca76c25bcb 100644 --- a/lib/glob.c +++ b/lib/glob.c @@ -1,5 +1,7 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) #include #include +#include /* * The only reason this code can be compiled as a module is because the @@ -20,7 +22,7 @@ MODULE_LICENSE("Dual MIT/GPL"); * Pattern metacharacters are ?, *, [ and \. * (And, inside character classes, !, - and ].) * - * This is small and simple implementation intended for device blacklists + * This is a small and simple implementation intended for device denylists * where a string is matched against a number of patterns. Thus, it * does not preprocess the patterns. It is non-recursive, and run-time * is at most quadratic: strlen(@str)*strlen(@pat). @@ -45,7 +47,7 @@ bool __pure glob_match(char const *pat, char const *str) * (no exception for /), it can be easily proved that there's * never a need to backtrack multiple levels. */ - char const *back_pat = NULL, *back_str; + char const *back_pat = NULL, *back_str = NULL; /* * Loop over each token (character or class) in pat, matching @@ -71,7 +73,7 @@ bool __pure glob_match(char const *pat, char const *str) if (c == '\0') /* No possible match */ return false; bool match = false, inverted = (*pat == '!'); - char const *class = pat + inverted; + char const *class = inverted ? pat + 1 : pat; unsigned char a = *class++; /* @@ -94,7 +96,8 @@ bool __pure glob_match(char const *pat, char const *str) class += 2; /* Any special action if a > b? */ } - match |= (a <= c && c <= b); + if (a <= c && c <= b) + match = true; } while ((a = *class++) != ']'); if (match == inverted) diff --git a/lib/inflate.c b/lib/inflate.c index eab886baa1b4..44a7da582baa 100644 --- a/lib/inflate.c +++ b/lib/inflate.c @@ -9,7 +9,7 @@ * based on gzip-1.0.3 * * Nicolas Pitre , 1999/04/14 : - * Little mods for all variable to reside either into rodata or bss segments + * Little mods for all variables to reside either into rodata or bss segments * by marking constant variables with 'const' and initializing all the others * at run-time only. This allows for the kernel uncompressor to run * directly from Flash or ROM memory on embedded systems. @@ -286,7 +286,7 @@ static void free(void *where) the longer codes. The time it costs to decode the longer codes is then traded against the time it takes to make longer tables. - This results of this trade are in the variables lbits and dbits + The results of this trade are in the variables lbits and dbits below. lbits is the number of bits the first level table for literal/ length codes can decode in one step, and dbits is the same thing for the distance codes. Subsequent tables are also less than or equal to @@ -811,6 +811,8 @@ DEBG(""); - ret = 0; out: free(ll); return ret; diff --git a/lib/list_sort.c b/lib/list_sort.c index a310ecb7ccc0..ff99203f208f 100644 --- a/lib/list_sort.c +++ b/lib/list_sort.c @@ -50,7 +50,6 @@ static void merge_final(void *priv, list_cmp_func_t cmp, struct list_head *head, struct list_head *a, struct list_head *b) { struct list_head *tail = head; - u8 count = 0; for (;;) { /* if equal, take 'a' -- important for sort stability */ @@ -76,15 +75,6 @@ static void merge_final(void *priv, list_cmp_func_t cmp, struct list_head *head, /* Finish linking remainder of list b on to tail */ tail->next = b; do { - /* - * If the merge is highly unbalanced (e.g. the input is - * already sorted), this loop may run many iterations. - * Continue callbacks to the client even though no - * element comparison is needed, so the client's cmp() - * routine can invoke cond_resched() periodically. - */ - if (unlikely(!++count)) - cmp(priv, b, b); b->prev = tail; tail = b; b = b->next; diff --git a/lib/math/Kconfig b/lib/math/Kconfig index 0634b428d0cb..0e6d9cffc5d6 100644 --- a/lib/math/Kconfig +++ b/lib/math/Kconfig @@ -5,6 +5,9 @@ config CORDIC This option provides an implementation of the CORDIC algorithm; calculations are in fixed point. Module will be called cordic. +config POLYNOMIAL + tristate + config PRIME_NUMBERS tristate "Simple prime number generator for testing" help diff --git a/lib/math/Makefile b/lib/math/Makefile index d1caba23baa0..9a3850d55b79 100644 --- a/lib/math/Makefile +++ b/lib/math/Makefile @@ -2,6 +2,7 @@ obj-y += div64.o gcd.o lcm.o int_log.o int_pow.o int_sqrt.o reciprocal_div.o obj-$(CONFIG_CORDIC) += cordic.o +obj-$(CONFIG_POLYNOMIAL) += polynomial.o obj-$(CONFIG_PRIME_NUMBERS) += prime_numbers.o obj-$(CONFIG_RATIONAL) += rational.o diff --git a/lib/polynomial.c b/lib/math/polynomial.c similarity index 88% rename from lib/polynomial.c rename to lib/math/polynomial.c index 66d383445fec..f26677cfeeff 100644 --- a/lib/polynomial.c +++ b/lib/math/polynomial.c @@ -10,21 +10,19 @@ * */ -#include +#include +#include #include #include /* - * Originally this was part of drivers/hwmon/bt1-pvt.c. - * There the following conversion is used and should serve as an example here: + * The following conversion is an example: * * The original translation formulae of the temperature (in degrees of Celsius) * to PVT data and vice-versa are following: * - * N = 1.8322e-8*(T^4) + 2.343e-5*(T^3) + 8.7018e-3*(T^2) + 3.9269*(T^1) + - * 1.7204e2 - * T = -1.6743e-11*(N^4) + 8.1542e-8*(N^3) + -1.8201e-4*(N^2) + - * 3.1020e-1*(N^1) - 4.838e1 + * N = 1.8322e-8*(T^4) + 2.343e-5*(T^3) + 8.7018e-3*(T^2) + 3.9269*(T^1) + 1.7204e2 + * T = -1.6743e-11*(N^4) + 8.1542e-8*(N^3) + -1.8201e-4*(N^2) + 3.1020e-1*(N^1) - 4.838e1 * * where T = [-48.380, 147.438]C and N = [0, 1023]. * @@ -35,10 +33,9 @@ * formulae to accept millidegrees of Celsius. Here what they look like after * the alterations: * - * N = (18322e-20*(T^4) + 2343e-13*(T^3) + 87018e-9*(T^2) + 39269e-3*T + - * 17204e2) / 1e4 - * T = -16743e-12*(D^4) + 81542e-9*(D^3) - 182010e-6*(D^2) + 310200e-3*D - - * 48380 + * N = (18322e-20*(T^4) + 2343e-13*(T^3) + 87018e-9*(T^2) + 39269e-3*T + 17204e2) / 1e4 + * T = -16743e-12*(D^4) + 81542e-9*(D^3) - 182010e-6*(D^2) + 310200e-3*D - 48380 + * * where T = [-48380, 147438] mC and N = [0, 1023]. * * static const struct polynomial poly_temp_to_N = { @@ -68,13 +65,13 @@ * polynomial_calc - calculate a polynomial using integer arithmetic * * @poly: pointer to the descriptor of the polynomial - * @data: input value of the polynimal + * @data: input value of the polynomial * * Calculate the result of a polynomial using only integer arithmetic. For * this to work without too much loss of precision the coefficients has to * be altered. This is called factor redistribution. * - * Returns the result of the polynomial calculation. + * Return: the result of the polynomial calculation. */ long polynomial_calc(const struct polynomial *poly, long data) { diff --git a/lib/parser.c b/lib/parser.c index 73e8f8e5be73..62da0ac0d438 100644 --- a/lib/parser.c +++ b/lib/parser.c @@ -315,7 +315,7 @@ bool match_wildcard(const char *pattern, const char *str) } } - if (*p == '*') + while (*p == '*') ++p; return !*p; } diff --git a/lib/raid/.kunitconfig b/lib/raid/.kunitconfig new file mode 100644 index 000000000000..351d22ed1954 --- /dev/null +++ b/lib/raid/.kunitconfig @@ -0,0 +1,3 @@ +CONFIG_KUNIT=y +CONFIG_BTRFS_FS=y +CONFIG_XOR_KUNIT_TEST=y diff --git a/lib/raid/Kconfig b/lib/raid/Kconfig new file mode 100644 index 000000000000..5ab2b0a7be4c --- /dev/null +++ b/lib/raid/Kconfig @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: GPL-2.0 + +config XOR_BLOCKS + tristate + +# selected by architectures that provide an optimized XOR implementation +config XOR_BLOCKS_ARCH + depends on XOR_BLOCKS + default y if ALPHA + default y if ARM + default y if ARM64 + default y if CPU_HAS_LSX # loongarch + default y if ALTIVEC # powerpc + default y if RISCV_ISA_V + default y if SPARC + default y if S390 + default y if X86_32 + default y if X86_64 + bool + +config XOR_KUNIT_TEST + tristate "KUnit tests for xor_gen" if !KUNIT_ALL_TESTS + depends on KUNIT + depends on XOR_BLOCKS + default KUNIT_ALL_TESTS + help + Unit tests for the XOR library functions. + + This is intended to help people writing architecture-specific + optimized versions. If unsure, say N. diff --git a/lib/raid/Makefile b/lib/raid/Makefile new file mode 100644 index 000000000000..3540fe846dc4 --- /dev/null +++ b/lib/raid/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-y += xor/ diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile new file mode 100644 index 000000000000..4d633dfd5b90 --- /dev/null +++ b/lib/raid/xor/Makefile @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: GPL-2.0 + +ccflags-y += -I $(src) + +obj-$(CONFIG_XOR_BLOCKS) += xor.o + +xor-y += xor-core.o +xor-y += xor-8regs.o +xor-y += xor-32regs.o +xor-y += xor-8regs-prefetch.o +xor-y += xor-32regs-prefetch.o + +ifeq ($(CONFIG_XOR_BLOCKS_ARCH),y) +CFLAGS_xor-core.o += -I$(src)/$(SRCARCH) +endif + +xor-$(CONFIG_ALPHA) += alpha/xor.o +xor-$(CONFIG_ARM) += arm/xor.o +ifeq ($(CONFIG_ARM),y) +xor-$(CONFIG_KERNEL_MODE_NEON) += arm/xor-neon.o arm/xor-neon-glue.o +endif +xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o +xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o +xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd_glue.o +xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o +xor-$(CONFIG_RISCV_ISA_V) += riscv/xor.o riscv/xor-glue.o +xor-$(CONFIG_SPARC32) += sparc/xor-sparc32.o +xor-$(CONFIG_SPARC64) += sparc/xor-sparc64.o sparc/xor-sparc64-glue.o +xor-$(CONFIG_S390) += s390/xor.o +xor-$(CONFIG_X86_32) += x86/xor-avx.o x86/xor-sse.o x86/xor-mmx.o +xor-$(CONFIG_X86_64) += x86/xor-avx.o x86/xor-sse.o +obj-y += tests/ + +CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU) +CFLAGS_REMOVE_arm/xor-neon.o += $(CC_FLAGS_NO_FPU) + +CFLAGS_arm64/xor-neon.o += $(CC_FLAGS_FPU) +CFLAGS_REMOVE_arm64/xor-neon.o += $(CC_FLAGS_NO_FPU) + +CFLAGS_powerpc/xor_vmx.o += -mhard-float -maltivec \ + $(call cc-option,-mabi=altivec) \ + -isystem $(shell $(CC) -print-file-name=include) diff --git a/arch/alpha/include/asm/xor.h b/lib/raid/xor/alpha/xor.c similarity index 95% rename from arch/alpha/include/asm/xor.h rename to lib/raid/xor/alpha/xor.c index e0de0c233ab9..a8f72f2dd3a5 100644 --- a/arch/alpha/include/asm/xor.h +++ b/lib/raid/xor/alpha/xor.c @@ -1,9 +1,9 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ +// SPDX-License-Identifier: GPL-2.0-or-later /* - * include/asm-alpha/xor.h - * - * Optimized RAID-5 checksumming functions for alpha EV5 and EV6 + * Optimized XOR parity functions for alpha EV5 and EV6 */ +#include "xor_impl.h" +#include "xor_arch.h" extern void xor_alpha_2(unsigned long bytes, unsigned long * __restrict p1, @@ -832,35 +832,17 @@ xor_alpha_prefetch_5: \n\ .end xor_alpha_prefetch_5 \n\ "); -static struct xor_block_template xor_block_alpha = { - .name = "alpha", - .do_2 = xor_alpha_2, - .do_3 = xor_alpha_3, - .do_4 = xor_alpha_4, - .do_5 = xor_alpha_5, +DO_XOR_BLOCKS(alpha, xor_alpha_2, xor_alpha_3, xor_alpha_4, xor_alpha_5); + +struct xor_block_template xor_block_alpha = { + .name = "alpha", + .xor_gen = xor_gen_alpha, }; -static struct xor_block_template xor_block_alpha_prefetch = { - .name = "alpha prefetch", - .do_2 = xor_alpha_prefetch_2, - .do_3 = xor_alpha_prefetch_3, - .do_4 = xor_alpha_prefetch_4, - .do_5 = xor_alpha_prefetch_5, +DO_XOR_BLOCKS(alpha_prefetch, xor_alpha_prefetch_2, xor_alpha_prefetch_3, + xor_alpha_prefetch_4, xor_alpha_prefetch_5); + +struct xor_block_template xor_block_alpha_prefetch = { + .name = "alpha prefetch", + .xor_gen = xor_gen_alpha_prefetch, }; - -/* For grins, also test the generic routines. */ -#include - -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ - do { \ - xor_speed(&xor_block_8regs); \ - xor_speed(&xor_block_32regs); \ - xor_speed(&xor_block_alpha); \ - xor_speed(&xor_block_alpha_prefetch); \ - } while (0) - -/* Force the use of alpha_prefetch if EV6, as it is significantly - faster in the cold cache case. */ -#define XOR_SELECT_TEMPLATE(FASTEST) \ - (implver() == IMPLVER_EV6 ? &xor_block_alpha_prefetch : FASTEST) diff --git a/lib/raid/xor/alpha/xor_arch.h b/lib/raid/xor/alpha/xor_arch.h new file mode 100644 index 000000000000..0dcfea578a48 --- /dev/null +++ b/lib/raid/xor/alpha/xor_arch.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#include + +extern struct xor_block_template xor_block_alpha; +extern struct xor_block_template xor_block_alpha_prefetch; + +/* + * Force the use of alpha_prefetch if EV6, as it is significantly faster in the + * cold cache case. + */ +static __always_inline void __init arch_xor_init(void) +{ + if (implver() == IMPLVER_EV6) { + xor_force(&xor_block_alpha_prefetch); + } else { + xor_register(&xor_block_8regs); + xor_register(&xor_block_32regs); + xor_register(&xor_block_alpha); + xor_register(&xor_block_alpha_prefetch); + } +} diff --git a/lib/raid/xor/arm/xor-neon-glue.c b/lib/raid/xor/arm/xor-neon-glue.c new file mode 100644 index 000000000000..cea39e019904 --- /dev/null +++ b/lib/raid/xor/arm/xor-neon-glue.c @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2001 Russell King + */ +#include "xor_impl.h" +#include "xor_arch.h" + +static void xor_gen_neon(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + kernel_neon_begin(); + xor_gen_neon_inner(dest, srcs, src_cnt, bytes); + kernel_neon_end(); +} + +struct xor_block_template xor_block_neon = { + .name = "neon", + .xor_gen = xor_gen_neon, +}; diff --git a/arch/arm/lib/xor-neon.c b/lib/raid/xor/arm/xor-neon.c similarity index 53% rename from arch/arm/lib/xor-neon.c rename to lib/raid/xor/arm/xor-neon.c index cf57fca97908..23147e3a7904 100644 --- a/arch/arm/lib/xor-neon.c +++ b/lib/raid/xor/arm/xor-neon.c @@ -1,15 +1,10 @@ // SPDX-License-Identifier: GPL-2.0-only /* - * linux/arch/arm/lib/xor-neon.c - * * Copyright (C) 2013 Linaro Ltd */ -#include -#include - -MODULE_DESCRIPTION("NEON accelerated XOR implementation"); -MODULE_LICENSE("GPL"); +#include "xor_impl.h" +#include "xor_arch.h" #ifndef __ARM_NEON__ #error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon' @@ -25,14 +20,7 @@ MODULE_LICENSE("GPL"); #pragma GCC optimize "tree-vectorize" #endif -#pragma GCC diagnostic ignored "-Wunused-variable" -#include +#define NO_TEMPLATE +#include "../xor-8regs.c" -struct xor_block_template const xor_block_neon_inner = { - .name = "__inner_neon__", - .do_2 = xor_8regs_2, - .do_3 = xor_8regs_3, - .do_4 = xor_8regs_4, - .do_5 = xor_8regs_5, -}; -EXPORT_SYMBOL(xor_block_neon_inner); +__DO_XOR_BLOCKS(neon_inner, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5); diff --git a/arch/arm/include/asm/xor.h b/lib/raid/xor/arm/xor.c similarity index 59% rename from arch/arm/include/asm/xor.h rename to lib/raid/xor/arm/xor.c index 934b549905f5..45139b6c55ea 100644 --- a/arch/arm/include/asm/xor.h +++ b/lib/raid/xor/arm/xor.c @@ -1,13 +1,9 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ +// SPDX-License-Identifier: GPL-2.0-only /* - * arch/arm/include/asm/xor.h - * * Copyright (C) 2001 Russell King */ -#include -#include -#include -#include +#include "xor_impl.h" +#include "xor_arch.h" #define __XOR(a1, a2) a1 ^= a2 @@ -131,95 +127,10 @@ xor_arm4regs_5(unsigned long bytes, unsigned long * __restrict p1, } while (--lines); } -static struct xor_block_template xor_block_arm4regs = { - .name = "arm4regs", - .do_2 = xor_arm4regs_2, - .do_3 = xor_arm4regs_3, - .do_4 = xor_arm4regs_4, - .do_5 = xor_arm4regs_5, +DO_XOR_BLOCKS(arm4regs, xor_arm4regs_2, xor_arm4regs_3, xor_arm4regs_4, + xor_arm4regs_5); + +struct xor_block_template xor_block_arm4regs = { + .name = "arm4regs", + .xor_gen = xor_gen_arm4regs, }; - -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ - do { \ - xor_speed(&xor_block_arm4regs); \ - xor_speed(&xor_block_8regs); \ - xor_speed(&xor_block_32regs); \ - NEON_TEMPLATES; \ - } while (0) - -#ifdef CONFIG_KERNEL_MODE_NEON - -extern struct xor_block_template const xor_block_neon_inner; - -static void -xor_neon_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2) -{ - if (in_interrupt()) { - xor_arm4regs_2(bytes, p1, p2); - } else { - kernel_neon_begin(); - xor_block_neon_inner.do_2(bytes, p1, p2); - kernel_neon_end(); - } -} - -static void -xor_neon_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3) -{ - if (in_interrupt()) { - xor_arm4regs_3(bytes, p1, p2, p3); - } else { - kernel_neon_begin(); - xor_block_neon_inner.do_3(bytes, p1, p2, p3); - kernel_neon_end(); - } -} - -static void -xor_neon_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4) -{ - if (in_interrupt()) { - xor_arm4regs_4(bytes, p1, p2, p3, p4); - } else { - kernel_neon_begin(); - xor_block_neon_inner.do_4(bytes, p1, p2, p3, p4); - kernel_neon_end(); - } -} - -static void -xor_neon_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5) -{ - if (in_interrupt()) { - xor_arm4regs_5(bytes, p1, p2, p3, p4, p5); - } else { - kernel_neon_begin(); - xor_block_neon_inner.do_5(bytes, p1, p2, p3, p4, p5); - kernel_neon_end(); - } -} - -static struct xor_block_template xor_block_neon = { - .name = "neon", - .do_2 = xor_neon_2, - .do_3 = xor_neon_3, - .do_4 = xor_neon_4, - .do_5 = xor_neon_5 -}; - -#define NEON_TEMPLATES \ - do { if (cpu_has_neon()) xor_speed(&xor_block_neon); } while (0) -#else -#define NEON_TEMPLATES -#endif diff --git a/lib/raid/xor/arm/xor_arch.h b/lib/raid/xor/arm/xor_arch.h new file mode 100644 index 000000000000..775ff835df65 --- /dev/null +++ b/lib/raid/xor/arm/xor_arch.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2001 Russell King + */ +#include + +extern struct xor_block_template xor_block_arm4regs; +extern struct xor_block_template xor_block_neon; + +void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes); + +static __always_inline void __init arch_xor_init(void) +{ + xor_register(&xor_block_arm4regs); + xor_register(&xor_block_8regs); + xor_register(&xor_block_32regs); +#ifdef CONFIG_KERNEL_MODE_NEON + if (cpu_has_neon()) + xor_register(&xor_block_neon); +#endif +} diff --git a/lib/raid/xor/arm64/xor-neon-glue.c b/lib/raid/xor/arm64/xor-neon-glue.c new file mode 100644 index 000000000000..f0284f86feb4 --- /dev/null +++ b/lib/raid/xor/arm64/xor-neon-glue.c @@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Authors: Jackie Liu + * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd. + */ + +#include +#include "xor_impl.h" +#include "xor_arch.h" +#include "xor-neon.h" + +#define XOR_TEMPLATE(_name) \ +static void xor_gen_##_name(void *dest, void **srcs, unsigned int src_cnt, \ + unsigned int bytes) \ +{ \ + scoped_ksimd() \ + xor_gen_##_name##_inner(dest, srcs, src_cnt, bytes); \ +} \ + \ +struct xor_block_template xor_block_##_name = { \ + .name = __stringify(_name), \ + .xor_gen = xor_gen_##_name, \ +}; + +XOR_TEMPLATE(neon); +XOR_TEMPLATE(eor3); diff --git a/arch/arm64/lib/xor-neon.c b/lib/raid/xor/arm64/xor-neon.c similarity index 76% rename from arch/arm64/lib/xor-neon.c rename to lib/raid/xor/arm64/xor-neon.c index 8fffebfa17b2..97ef3cb92496 100644 --- a/arch/arm64/lib/xor-neon.c +++ b/lib/raid/xor/arm64/xor-neon.c @@ -1,17 +1,17 @@ // SPDX-License-Identifier: GPL-2.0-only /* - * arch/arm64/lib/xor-neon.c - * * Authors: Jackie Liu * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd. */ -#include -#include +#include #include +#include "xor_impl.h" +#include "xor_arch.h" +#include "xor-neon.h" -static void xor_arm64_neon_2(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2) +static void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2) { uint64_t *dp1 = (uint64_t *)p1; uint64_t *dp2 = (uint64_t *)p2; @@ -37,9 +37,9 @@ static void xor_arm64_neon_2(unsigned long bytes, unsigned long * __restrict p1, } while (--lines > 0); } -static void xor_arm64_neon_3(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3) +static void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3) { uint64_t *dp1 = (uint64_t *)p1; uint64_t *dp2 = (uint64_t *)p2; @@ -73,10 +73,10 @@ static void xor_arm64_neon_3(unsigned long bytes, unsigned long * __restrict p1, } while (--lines > 0); } -static void xor_arm64_neon_4(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4) +static void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4) { uint64_t *dp1 = (uint64_t *)p1; uint64_t *dp2 = (uint64_t *)p2; @@ -118,11 +118,11 @@ static void xor_arm64_neon_4(unsigned long bytes, unsigned long * __restrict p1, } while (--lines > 0); } -static void xor_arm64_neon_5(unsigned long bytes, unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5) +static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4, + const unsigned long * __restrict p5) { uint64_t *dp1 = (uint64_t *)p1; uint64_t *dp2 = (uint64_t *)p2; @@ -172,14 +172,8 @@ static void xor_arm64_neon_5(unsigned long bytes, unsigned long * __restrict p1, } while (--lines > 0); } -struct xor_block_template xor_block_inner_neon __ro_after_init = { - .name = "__inner_neon__", - .do_2 = xor_arm64_neon_2, - .do_3 = xor_arm64_neon_3, - .do_4 = xor_arm64_neon_4, - .do_5 = xor_arm64_neon_5, -}; -EXPORT_SYMBOL(xor_block_inner_neon); +__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4, + __xor_neon_5); static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r) { @@ -191,10 +185,9 @@ static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r) return res; } -static void xor_arm64_eor3_3(unsigned long bytes, - unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3) +static void __xor_eor3_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3) { uint64_t *dp1 = (uint64_t *)p1; uint64_t *dp2 = (uint64_t *)p2; @@ -226,11 +219,10 @@ static void xor_arm64_eor3_3(unsigned long bytes, } while (--lines > 0); } -static void xor_arm64_eor3_4(unsigned long bytes, - unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4) +static void __xor_eor3_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4) { uint64_t *dp1 = (uint64_t *)p1; uint64_t *dp2 = (uint64_t *)p2; @@ -270,12 +262,11 @@ static void xor_arm64_eor3_4(unsigned long bytes, } while (--lines > 0); } -static void xor_arm64_eor3_5(unsigned long bytes, - unsigned long * __restrict p1, - const unsigned long * __restrict p2, - const unsigned long * __restrict p3, - const unsigned long * __restrict p4, - const unsigned long * __restrict p5) +static void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4, + const unsigned long * __restrict p5) { uint64_t *dp1 = (uint64_t *)p1; uint64_t *dp2 = (uint64_t *)p2; @@ -317,22 +308,5 @@ static void xor_arm64_eor3_5(unsigned long bytes, } while (--lines > 0); } -static int __init xor_neon_init(void) -{ - if (cpu_have_named_feature(SHA3)) { - xor_block_inner_neon.do_3 = xor_arm64_eor3_3; - xor_block_inner_neon.do_4 = xor_arm64_eor3_4; - xor_block_inner_neon.do_5 = xor_arm64_eor3_5; - } - return 0; -} -module_init(xor_neon_init); - -static void __exit xor_neon_exit(void) -{ -} -module_exit(xor_neon_exit); - -MODULE_AUTHOR("Jackie Liu "); -MODULE_DESCRIPTION("ARMv8 XOR Extensions"); -MODULE_LICENSE("GPL"); +__DO_XOR_BLOCKS(eor3_inner, __xor_neon_2, __xor_eor3_3, __xor_eor3_4, + __xor_eor3_5); diff --git a/lib/raid/xor/arm64/xor-neon.h b/lib/raid/xor/arm64/xor-neon.h new file mode 100644 index 000000000000..514699ba8f5f --- /dev/null +++ b/lib/raid/xor/arm64/xor-neon.h @@ -0,0 +1,6 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes); +void xor_gen_eor3_inner(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes); diff --git a/lib/raid/xor/arm64/xor_arch.h b/lib/raid/xor/arm64/xor_arch.h new file mode 100644 index 000000000000..5dbb40319501 --- /dev/null +++ b/lib/raid/xor/arm64/xor_arch.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Authors: Jackie Liu + * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd. + */ +#include + +extern struct xor_block_template xor_block_neon; +extern struct xor_block_template xor_block_eor3; + +static __always_inline void __init arch_xor_init(void) +{ + xor_register(&xor_block_8regs); + xor_register(&xor_block_32regs); + if (cpu_has_neon()) { + if (cpu_have_named_feature(SHA3)) + xor_register(&xor_block_eor3); + else + xor_register(&xor_block_neon); + } +} diff --git a/lib/raid/xor/loongarch/xor_arch.h b/lib/raid/xor/loongarch/xor_arch.h new file mode 100644 index 000000000000..fe5e8244fd0e --- /dev/null +++ b/lib/raid/xor/loongarch/xor_arch.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2023 WANG Xuerui + */ +#include + +/* + * For grins, also test the generic routines. + * + * More importantly: it cannot be ruled out at this point of time, that some + * future (maybe reduced) models could run the vector algorithms slower than + * the scalar ones, maybe for errata or micro-op reasons. It may be + * appropriate to revisit this after one or two more uarch generations. + */ + +extern struct xor_block_template xor_block_lsx; +extern struct xor_block_template xor_block_lasx; + +static __always_inline void __init arch_xor_init(void) +{ + xor_register(&xor_block_8regs); + xor_register(&xor_block_8regs_p); + xor_register(&xor_block_32regs); + xor_register(&xor_block_32regs_p); +#ifdef CONFIG_CPU_HAS_LSX + if (cpu_has_lsx) + xor_register(&xor_block_lsx); +#endif +#ifdef CONFIG_CPU_HAS_LASX + if (cpu_has_lasx) + xor_register(&xor_block_lasx); +#endif +} diff --git a/arch/loongarch/lib/xor_simd.c b/lib/raid/xor/loongarch/xor_simd.c similarity index 100% rename from arch/loongarch/lib/xor_simd.c rename to lib/raid/xor/loongarch/xor_simd.c diff --git a/arch/loongarch/lib/xor_simd.h b/lib/raid/xor/loongarch/xor_simd.h similarity index 100% rename from arch/loongarch/lib/xor_simd.h rename to lib/raid/xor/loongarch/xor_simd.h diff --git a/lib/raid/xor/loongarch/xor_simd_glue.c b/lib/raid/xor/loongarch/xor_simd_glue.c new file mode 100644 index 000000000000..7f324d924f87 --- /dev/null +++ b/lib/raid/xor/loongarch/xor_simd_glue.c @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * LoongArch SIMD XOR operations + * + * Copyright (C) 2023 WANG Xuerui + */ + +#include +#include +#include "xor_impl.h" +#include "xor_arch.h" +#include "xor_simd.h" + +#define MAKE_XOR_GLUES(flavor) \ +DO_XOR_BLOCKS(flavor##_inner, __xor_##flavor##_2, __xor_##flavor##_3, \ + __xor_##flavor##_4, __xor_##flavor##_5); \ + \ +static void xor_gen_##flavor(void *dest, void **srcs, unsigned int src_cnt, \ + unsigned int bytes) \ +{ \ + kernel_fpu_begin(); \ + xor_gen_##flavor##_inner(dest, srcs, src_cnt, bytes); \ + kernel_fpu_end(); \ +} \ + \ +struct xor_block_template xor_block_##flavor = { \ + .name = __stringify(flavor), \ + .xor_gen = xor_gen_##flavor \ +} + +#ifdef CONFIG_CPU_HAS_LSX +MAKE_XOR_GLUES(lsx); +#endif /* CONFIG_CPU_HAS_LSX */ + +#ifdef CONFIG_CPU_HAS_LASX +MAKE_XOR_GLUES(lasx); +#endif /* CONFIG_CPU_HAS_LASX */ diff --git a/arch/loongarch/lib/xor_template.c b/lib/raid/xor/loongarch/xor_template.c similarity index 100% rename from arch/loongarch/lib/xor_template.c rename to lib/raid/xor/loongarch/xor_template.c diff --git a/lib/raid/xor/powerpc/xor_arch.h b/lib/raid/xor/powerpc/xor_arch.h new file mode 100644 index 000000000000..3b00a4a2fd67 --- /dev/null +++ b/lib/raid/xor/powerpc/xor_arch.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * + * Copyright (C) IBM Corporation, 2012 + * + * Author: Anton Blanchard + */ +#include + +extern struct xor_block_template xor_block_altivec; + +static __always_inline void __init arch_xor_init(void) +{ + xor_register(&xor_block_8regs); + xor_register(&xor_block_8regs_p); + xor_register(&xor_block_32regs); + xor_register(&xor_block_32regs_p); +#ifdef CONFIG_ALTIVEC + if (cpu_has_feature(CPU_FTR_ALTIVEC)) + xor_register(&xor_block_altivec); +#endif +} diff --git a/arch/powerpc/lib/xor_vmx.c b/lib/raid/xor/powerpc/xor_vmx.c similarity index 69% rename from arch/powerpc/lib/xor_vmx.c rename to lib/raid/xor/powerpc/xor_vmx.c index aab49d056d18..09bed98c1bc7 100644 --- a/arch/powerpc/lib/xor_vmx.c +++ b/lib/raid/xor/powerpc/xor_vmx.c @@ -10,6 +10,7 @@ * Sparse (as at v0.5.0) gets very, very confused by this file. * Make it a bit simpler for it. */ +#include "xor_impl.h" #if !defined(__CHECKER__) #include #else @@ -49,9 +50,9 @@ typedef vector signed char unative_t; V1##_3 = vec_xor(V1##_3, V2##_3); \ } while (0) -void __xor_altivec_2(unsigned long bytes, - unsigned long * __restrict v1_in, - const unsigned long * __restrict v2_in) +static void __xor_altivec_2(unsigned long bytes, + unsigned long * __restrict v1_in, + const unsigned long * __restrict v2_in) { DEFINE(v1); DEFINE(v2); @@ -68,10 +69,10 @@ void __xor_altivec_2(unsigned long bytes, } while (--lines > 0); } -void __xor_altivec_3(unsigned long bytes, - unsigned long * __restrict v1_in, - const unsigned long * __restrict v2_in, - const unsigned long * __restrict v3_in) +static void __xor_altivec_3(unsigned long bytes, + unsigned long * __restrict v1_in, + const unsigned long * __restrict v2_in, + const unsigned long * __restrict v3_in) { DEFINE(v1); DEFINE(v2); @@ -92,11 +93,11 @@ void __xor_altivec_3(unsigned long bytes, } while (--lines > 0); } -void __xor_altivec_4(unsigned long bytes, - unsigned long * __restrict v1_in, - const unsigned long * __restrict v2_in, - const unsigned long * __restrict v3_in, - const unsigned long * __restrict v4_in) +static void __xor_altivec_4(unsigned long bytes, + unsigned long * __restrict v1_in, + const unsigned long * __restrict v2_in, + const unsigned long * __restrict v3_in, + const unsigned long * __restrict v4_in) { DEFINE(v1); DEFINE(v2); @@ -121,12 +122,12 @@ void __xor_altivec_4(unsigned long bytes, } while (--lines > 0); } -void __xor_altivec_5(unsigned long bytes, - unsigned long * __restrict v1_in, - const unsigned long * __restrict v2_in, - const unsigned long * __restrict v3_in, - const unsigned long * __restrict v4_in, - const unsigned long * __restrict v5_in) +static void __xor_altivec_5(unsigned long bytes, + unsigned long * __restrict v1_in, + const unsigned long * __restrict v2_in, + const unsigned long * __restrict v3_in, + const unsigned long * __restrict v4_in, + const unsigned long * __restrict v5_in) { DEFINE(v1); DEFINE(v2); @@ -154,3 +155,6 @@ void __xor_altivec_5(unsigned long bytes, v5 += 4; } while (--lines > 0); } + +__DO_XOR_BLOCKS(altivec_inner, __xor_altivec_2, __xor_altivec_3, + __xor_altivec_4, __xor_altivec_5); diff --git a/lib/raid/xor/powerpc/xor_vmx.h b/lib/raid/xor/powerpc/xor_vmx.h new file mode 100644 index 000000000000..1d26c1133a86 --- /dev/null +++ b/lib/raid/xor/powerpc/xor_vmx.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Simple interface to link xor_vmx.c and xor_vmx_glue.c + * + * Separating these file ensures that no altivec instructions are run + * outside of the enable/disable altivec block. + */ + +void xor_gen_altivec_inner(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes); diff --git a/lib/raid/xor/powerpc/xor_vmx_glue.c b/lib/raid/xor/powerpc/xor_vmx_glue.c new file mode 100644 index 000000000000..dbfbb5cadc36 --- /dev/null +++ b/lib/raid/xor/powerpc/xor_vmx_glue.c @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Altivec XOR operations + * + * Copyright 2017 IBM Corp. + */ + +#include +#include +#include +#include "xor_impl.h" +#include "xor_arch.h" +#include "xor_vmx.h" + +static void xor_gen_altivec(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + preempt_disable(); + enable_kernel_altivec(); + xor_gen_altivec_inner(dest, srcs, src_cnt, bytes); + disable_kernel_altivec(); + preempt_enable(); +} + +struct xor_block_template xor_block_altivec = { + .name = "altivec", + .xor_gen = xor_gen_altivec, +}; diff --git a/lib/raid/xor/riscv/xor-glue.c b/lib/raid/xor/riscv/xor-glue.c new file mode 100644 index 000000000000..2e4c1b05d998 --- /dev/null +++ b/lib/raid/xor/riscv/xor-glue.c @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2021 SiFive + */ + +#include +#include +#include +#include "xor_impl.h" +#include "xor_arch.h" + +DO_XOR_BLOCKS(vector_inner, xor_regs_2_, xor_regs_3_, xor_regs_4_, xor_regs_5_); + +static void xor_gen_vector(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + kernel_vector_begin(); + xor_gen_vector_inner(dest, srcs, src_cnt, bytes); + kernel_vector_end(); +} + +struct xor_block_template xor_block_rvv = { + .name = "rvv", + .xor_gen = xor_gen_vector, +}; diff --git a/arch/riscv/lib/xor.S b/lib/raid/xor/riscv/xor.S similarity index 92% rename from arch/riscv/lib/xor.S rename to lib/raid/xor/riscv/xor.S index b28f2430e52f..56fb7fc1e2cd 100644 --- a/arch/riscv/lib/xor.S +++ b/lib/raid/xor/riscv/xor.S @@ -18,7 +18,6 @@ SYM_FUNC_START(xor_regs_2_) bnez a0, xor_regs_2_ ret SYM_FUNC_END(xor_regs_2_) -EXPORT_SYMBOL(xor_regs_2_) SYM_FUNC_START(xor_regs_3_) vsetvli a4, a0, e8, m8, ta, ma @@ -35,7 +34,6 @@ SYM_FUNC_START(xor_regs_3_) bnez a0, xor_regs_3_ ret SYM_FUNC_END(xor_regs_3_) -EXPORT_SYMBOL(xor_regs_3_) SYM_FUNC_START(xor_regs_4_) vsetvli a5, a0, e8, m8, ta, ma @@ -55,7 +53,6 @@ SYM_FUNC_START(xor_regs_4_) bnez a0, xor_regs_4_ ret SYM_FUNC_END(xor_regs_4_) -EXPORT_SYMBOL(xor_regs_4_) SYM_FUNC_START(xor_regs_5_) vsetvli a6, a0, e8, m8, ta, ma @@ -78,4 +75,3 @@ SYM_FUNC_START(xor_regs_5_) bnez a0, xor_regs_5_ ret SYM_FUNC_END(xor_regs_5_) -EXPORT_SYMBOL(xor_regs_5_) diff --git a/lib/raid/xor/riscv/xor_arch.h b/lib/raid/xor/riscv/xor_arch.h new file mode 100644 index 000000000000..9240857d760b --- /dev/null +++ b/lib/raid/xor/riscv/xor_arch.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ +#include + +extern struct xor_block_template xor_block_rvv; + +static __always_inline void __init arch_xor_init(void) +{ + xor_register(&xor_block_8regs); + xor_register(&xor_block_32regs); +#ifdef CONFIG_RISCV_ISA_V + if (has_vector()) + xor_register(&xor_block_rvv); +#endif +} diff --git a/arch/s390/lib/xor.c b/lib/raid/xor/s390/xor.c similarity index 93% rename from arch/s390/lib/xor.c rename to lib/raid/xor/s390/xor.c index 5363e4c2462d..0c478678a129 100644 --- a/arch/s390/lib/xor.c +++ b/lib/raid/xor/s390/xor.c @@ -7,9 +7,8 @@ */ #include -#include -#include -#include +#include "xor_impl.h" +#include "xor_arch.h" static void xor_xc_2(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2) @@ -126,11 +125,9 @@ static void xor_xc_5(unsigned long bytes, unsigned long * __restrict p1, : : "0", "cc", "memory"); } +DO_XOR_BLOCKS(xc, xor_xc_2, xor_xc_3, xor_xc_4, xor_xc_5); + struct xor_block_template xor_block_xc = { - .name = "xc", - .do_2 = xor_xc_2, - .do_3 = xor_xc_3, - .do_4 = xor_xc_4, - .do_5 = xor_xc_5, + .name = "xc", + .xor_gen = xor_gen_xc, }; -EXPORT_SYMBOL(xor_block_xc); diff --git a/lib/raid/xor/s390/xor_arch.h b/lib/raid/xor/s390/xor_arch.h new file mode 100644 index 000000000000..4a233ed2b97a --- /dev/null +++ b/lib/raid/xor/s390/xor_arch.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Optimited xor routines + * + * Copyright IBM Corp. 2016 + * Author(s): Martin Schwidefsky + */ +extern struct xor_block_template xor_block_xc; + +static __always_inline void __init arch_xor_init(void) +{ + xor_force(&xor_block_xc); +} diff --git a/arch/sparc/include/asm/xor_32.h b/lib/raid/xor/sparc/xor-sparc32.c similarity index 92% rename from arch/sparc/include/asm/xor_32.h rename to lib/raid/xor/sparc/xor-sparc32.c index 0351813cf3af..fb37631e90e6 100644 --- a/arch/sparc/include/asm/xor_32.h +++ b/lib/raid/xor/sparc/xor-sparc32.c @@ -1,16 +1,12 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * include/asm/xor.h - * - * Optimized RAID-5 checksumming functions for 32-bit Sparc. - */ - +// SPDX-License-Identifier: GPL-2.0-or-later /* * High speed xor_block operation for RAID4/5 utilizing the * ldd/std SPARC instructions. * * Copyright (C) 1999 Jakub Jelinek (jj@ultra.linux.cz) */ +#include "xor_impl.h" +#include "xor_arch.h" static void sparc_2(unsigned long bytes, unsigned long * __restrict p1, @@ -248,21 +244,9 @@ sparc_5(unsigned long bytes, unsigned long * __restrict p1, } while (--lines > 0); } -static struct xor_block_template xor_block_SPARC = { - .name = "SPARC", - .do_2 = sparc_2, - .do_3 = sparc_3, - .do_4 = sparc_4, - .do_5 = sparc_5, +DO_XOR_BLOCKS(sparc32, sparc_2, sparc_3, sparc_4, sparc_5); + +struct xor_block_template xor_block_SPARC = { + .name = "SPARC", + .xor_gen = xor_gen_sparc32, }; - -/* For grins, also test the generic routines. */ -#include - -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ - do { \ - xor_speed(&xor_block_8regs); \ - xor_speed(&xor_block_32regs); \ - xor_speed(&xor_block_SPARC); \ - } while (0) diff --git a/arch/sparc/include/asm/xor_64.h b/lib/raid/xor/sparc/xor-sparc64-glue.c similarity index 63% rename from arch/sparc/include/asm/xor_64.h rename to lib/raid/xor/sparc/xor-sparc64-glue.c index caaddea8ad79..a8a686e0d258 100644 --- a/arch/sparc/include/asm/xor_64.h +++ b/lib/raid/xor/sparc/xor-sparc64-glue.c @@ -1,7 +1,5 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ +// SPDX-License-Identifier: GPL-2.0-or-later /* - * include/asm/xor.h - * * High speed xor_block operation for RAID4/5 utilizing the * UltraSparc Visual Instruction Set and Niagara block-init * twin-load instructions. @@ -10,7 +8,8 @@ * Copyright (C) 2006 David S. Miller */ -#include +#include "xor_impl.h" +#include "xor_arch.h" void xor_vis_2(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2); @@ -29,12 +28,11 @@ void xor_vis_5(unsigned long bytes, unsigned long * __restrict p1, /* XXX Ugh, write cheetah versions... -DaveM */ -static struct xor_block_template xor_block_VIS = { - .name = "VIS", - .do_2 = xor_vis_2, - .do_3 = xor_vis_3, - .do_4 = xor_vis_4, - .do_5 = xor_vis_5, +DO_XOR_BLOCKS(vis, xor_vis_2, xor_vis_3, xor_vis_4, xor_vis_5); + +struct xor_block_template xor_block_VIS = { + .name = "VIS", + .xor_gen = xor_gen_vis, }; void xor_niagara_2(unsigned long bytes, unsigned long * __restrict p1, @@ -52,28 +50,10 @@ void xor_niagara_5(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p4, const unsigned long * __restrict p5); -static struct xor_block_template xor_block_niagara = { - .name = "Niagara", - .do_2 = xor_niagara_2, - .do_3 = xor_niagara_3, - .do_4 = xor_niagara_4, - .do_5 = xor_niagara_5, +DO_XOR_BLOCKS(niagara, xor_niagara_2, xor_niagara_3, xor_niagara_4, + xor_niagara_5); + +struct xor_block_template xor_block_niagara = { + .name = "Niagara", + .xor_gen = xor_gen_niagara, }; - -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ - do { \ - xor_speed(&xor_block_VIS); \ - xor_speed(&xor_block_niagara); \ - } while (0) - -/* For VIS for everything except Niagara. */ -#define XOR_SELECT_TEMPLATE(FASTEST) \ - ((tlb_type == hypervisor && \ - (sun4v_chip_type == SUN4V_CHIP_NIAGARA1 || \ - sun4v_chip_type == SUN4V_CHIP_NIAGARA2 || \ - sun4v_chip_type == SUN4V_CHIP_NIAGARA3 || \ - sun4v_chip_type == SUN4V_CHIP_NIAGARA4 || \ - sun4v_chip_type == SUN4V_CHIP_NIAGARA5)) ? \ - &xor_block_niagara : \ - &xor_block_VIS) diff --git a/arch/sparc/lib/xor.S b/lib/raid/xor/sparc/xor-sparc64.S similarity index 98% rename from arch/sparc/lib/xor.S rename to lib/raid/xor/sparc/xor-sparc64.S index 35461e3b2a9b..a7b74d473bd4 100644 --- a/arch/sparc/lib/xor.S +++ b/lib/raid/xor/sparc/xor-sparc64.S @@ -1,7 +1,5 @@ /* SPDX-License-Identifier: GPL-2.0 */ /* - * arch/sparc64/lib/xor.S - * * High speed xor_block operation for RAID4/5 utilizing the * UltraSparc Visual Instruction Set and Niagara store-init/twin-load. * @@ -92,7 +90,6 @@ ENTRY(xor_vis_2) retl wr %g0, 0, %fprs ENDPROC(xor_vis_2) -EXPORT_SYMBOL(xor_vis_2) ENTRY(xor_vis_3) rd %fprs, %o5 @@ -159,7 +156,6 @@ ENTRY(xor_vis_3) retl wr %g0, 0, %fprs ENDPROC(xor_vis_3) -EXPORT_SYMBOL(xor_vis_3) ENTRY(xor_vis_4) rd %fprs, %o5 @@ -245,7 +241,6 @@ ENTRY(xor_vis_4) retl wr %g0, 0, %fprs ENDPROC(xor_vis_4) -EXPORT_SYMBOL(xor_vis_4) ENTRY(xor_vis_5) save %sp, -192, %sp @@ -352,7 +347,6 @@ ENTRY(xor_vis_5) ret restore ENDPROC(xor_vis_5) -EXPORT_SYMBOL(xor_vis_5) /* Niagara versions. */ ENTRY(xor_niagara_2) /* %o0=bytes, %o1=dest, %o2=src */ @@ -399,7 +393,6 @@ ENTRY(xor_niagara_2) /* %o0=bytes, %o1=dest, %o2=src */ ret restore ENDPROC(xor_niagara_2) -EXPORT_SYMBOL(xor_niagara_2) ENTRY(xor_niagara_3) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2 */ save %sp, -192, %sp @@ -461,7 +454,6 @@ ENTRY(xor_niagara_3) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2 */ ret restore ENDPROC(xor_niagara_3) -EXPORT_SYMBOL(xor_niagara_3) ENTRY(xor_niagara_4) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2, %o4=src3 */ save %sp, -192, %sp @@ -544,7 +536,6 @@ ENTRY(xor_niagara_4) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2, %o4=src3 */ ret restore ENDPROC(xor_niagara_4) -EXPORT_SYMBOL(xor_niagara_4) ENTRY(xor_niagara_5) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2, %o4=src3, %o5=src4 */ save %sp, -192, %sp @@ -643,4 +634,3 @@ ENTRY(xor_niagara_5) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2, %o4=src3, %o5=s ret restore ENDPROC(xor_niagara_5) -EXPORT_SYMBOL(xor_niagara_5) diff --git a/lib/raid/xor/sparc/xor_arch.h b/lib/raid/xor/sparc/xor_arch.h new file mode 100644 index 000000000000..af288abe4e91 --- /dev/null +++ b/lib/raid/xor/sparc/xor_arch.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 1997, 1999 Jakub Jelinek (jj@ultra.linux.cz) + * Copyright (C) 2006 David S. Miller + */ +#if defined(__sparc__) && defined(__arch64__) +#include + +extern struct xor_block_template xor_block_VIS; +extern struct xor_block_template xor_block_niagara; + +static __always_inline void __init arch_xor_init(void) +{ + /* Force VIS for everything except Niagara. */ + if (tlb_type == hypervisor && + (sun4v_chip_type == SUN4V_CHIP_NIAGARA1 || + sun4v_chip_type == SUN4V_CHIP_NIAGARA2 || + sun4v_chip_type == SUN4V_CHIP_NIAGARA3 || + sun4v_chip_type == SUN4V_CHIP_NIAGARA4 || + sun4v_chip_type == SUN4V_CHIP_NIAGARA5)) + xor_force(&xor_block_niagara); + else + xor_force(&xor_block_VIS); +} +#else /* sparc64 */ + +extern struct xor_block_template xor_block_SPARC; + +static __always_inline void __init arch_xor_init(void) +{ + xor_register(&xor_block_8regs); + xor_register(&xor_block_32regs); + xor_register(&xor_block_SPARC); +} +#endif /* !sparc64 */ diff --git a/lib/raid/xor/tests/Makefile b/lib/raid/xor/tests/Makefile new file mode 100644 index 000000000000..661e8f6ffd1f --- /dev/null +++ b/lib/raid/xor/tests/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only + +obj-$(CONFIG_XOR_KUNIT_TEST) += xor_kunit.o diff --git a/lib/raid/xor/tests/xor_kunit.c b/lib/raid/xor/tests/xor_kunit.c new file mode 100644 index 000000000000..0c2a3a420bf9 --- /dev/null +++ b/lib/raid/xor/tests/xor_kunit.c @@ -0,0 +1,187 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Unit test the XOR library functions. + * + * Copyright 2024 Google LLC + * Copyright 2026 Christoph Hellwig + * + * Based on the CRC tests by Eric Biggers . + */ +#include +#include +#include +#include +#include + +#define XOR_KUNIT_SEED 42 +#define XOR_KUNIT_MAX_BYTES 16384 +#define XOR_KUNIT_MAX_BUFFERS 64 +#define XOR_KUNIT_NUM_TEST_ITERS 1000 + +static struct rnd_state rng; +static void *test_buffers[XOR_KUNIT_MAX_BUFFERS]; +static void *test_dest; +static void *test_ref; +static size_t test_buflen; + +static u32 rand32(void) +{ + return prandom_u32_state(&rng); +} + +/* Reference implementation using dumb byte-wise XOR */ +static void xor_ref(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + unsigned int off, idx; + u8 *d = dest; + + for (off = 0; off < bytes; off++) { + for (idx = 0; idx < src_cnt; idx++) { + u8 *src = srcs[idx]; + + d[off] ^= src[off]; + } + } +} + +/* Generate a random length that is a multiple of 512. */ +static unsigned int random_length(unsigned int max_length) +{ + return round_up((rand32() % max_length) + 1, 512); +} + +/* Generate a random alignment that is a multiple of 64. */ +static unsigned int random_alignment(unsigned int max_alignment) +{ + return ((rand32() % max_alignment) + 1) & ~63; +} + +static void xor_generate_random_data(void) +{ + int i; + + prandom_bytes_state(&rng, test_dest, test_buflen); + memcpy(test_ref, test_dest, test_buflen); + for (i = 0; i < XOR_KUNIT_MAX_BUFFERS; i++) + prandom_bytes_state(&rng, test_buffers[i], test_buflen); +} + +/* Test that xor_gen gives the same result as a reference implementation. */ +static void xor_test(struct kunit *test) +{ + void *aligned_buffers[XOR_KUNIT_MAX_BUFFERS]; + size_t i; + + for (i = 0; i < XOR_KUNIT_NUM_TEST_ITERS; i++) { + unsigned int nr_buffers = + (rand32() % XOR_KUNIT_MAX_BUFFERS) + 1; + unsigned int len = random_length(XOR_KUNIT_MAX_BYTES); + unsigned int max_alignment, align = 0; + void *buffers; + + if (rand32() % 8 == 0) + /* Refresh the data occasionally. */ + xor_generate_random_data(); + + /* + * If we're not using the entire buffer size, inject randomize + * alignment into the buffer. + */ + max_alignment = XOR_KUNIT_MAX_BYTES - len; + if (max_alignment == 0) { + buffers = test_buffers; + } else if (rand32() % 2 == 0) { + /* Use random alignments mod 64 */ + int j; + + for (j = 0; j < nr_buffers; j++) + aligned_buffers[j] = test_buffers[j] + + random_alignment(max_alignment); + buffers = aligned_buffers; + align = random_alignment(max_alignment); + } else { + /* Go up to the guard page, to catch buffer overreads */ + int j; + + align = test_buflen - len; + for (j = 0; j < nr_buffers; j++) + aligned_buffers[j] = test_buffers[j] + align; + buffers = aligned_buffers; + } + + /* + * Compute the XOR, and verify that it equals the XOR computed + * by a simple byte-at-a-time reference implementation. + */ + xor_ref(test_ref + align, buffers, nr_buffers, len); + xor_gen(test_dest + align, buffers, nr_buffers, len); + KUNIT_EXPECT_MEMEQ_MSG(test, test_ref + align, + test_dest + align, len, + "Wrong result with buffers=%u, len=%u, unaligned=%s, at_end=%s", + nr_buffers, len, + str_yes_no(max_alignment), + str_yes_no(align + len == test_buflen)); + } +} + +static struct kunit_case xor_test_cases[] = { + KUNIT_CASE(xor_test), + {}, +}; + +static int xor_suite_init(struct kunit_suite *suite) +{ + int i; + + /* + * Allocate the test buffer using vmalloc() with a page-aligned length + * so that it is immediately followed by a guard page. This allows + * buffer overreads to be detected, even in assembly code. + */ + test_buflen = round_up(XOR_KUNIT_MAX_BYTES, PAGE_SIZE); + test_ref = vmalloc(test_buflen); + if (!test_ref) + return -ENOMEM; + test_dest = vmalloc(test_buflen); + if (!test_dest) + goto out_free_ref; + for (i = 0; i < XOR_KUNIT_MAX_BUFFERS; i++) { + test_buffers[i] = vmalloc(test_buflen); + if (!test_buffers[i]) + goto out_free_buffers; + } + + prandom_seed_state(&rng, XOR_KUNIT_SEED); + xor_generate_random_data(); + return 0; + +out_free_buffers: + while (--i >= 0) + vfree(test_buffers[i]); + vfree(test_dest); +out_free_ref: + vfree(test_ref); + return -ENOMEM; +} + +static void xor_suite_exit(struct kunit_suite *suite) +{ + int i; + + vfree(test_ref); + vfree(test_dest); + for (i = 0; i < XOR_KUNIT_MAX_BUFFERS; i++) + vfree(test_buffers[i]); +} + +static struct kunit_suite xor_test_suite = { + .name = "xor", + .test_cases = xor_test_cases, + .suite_init = xor_suite_init, + .suite_exit = xor_suite_exit, +}; +kunit_test_suite(xor_test_suite); + +MODULE_DESCRIPTION("Unit test for the XOR library functions"); +MODULE_LICENSE("GPL"); diff --git a/lib/raid/xor/um/xor_arch.h b/lib/raid/xor/um/xor_arch.h new file mode 100644 index 000000000000..a33e57a26c5e --- /dev/null +++ b/lib/raid/xor/um/xor_arch.h @@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include <../x86/xor_arch.h> diff --git a/arch/x86/include/asm/xor_avx.h b/lib/raid/xor/x86/xor-avx.c similarity index 84% rename from arch/x86/include/asm/xor_avx.h rename to lib/raid/xor/x86/xor-avx.c index 7f81dd5897f4..f7777d7aa269 100644 --- a/arch/x86/include/asm/xor_avx.h +++ b/lib/raid/xor/x86/xor-avx.c @@ -1,18 +1,16 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -#ifndef _ASM_X86_XOR_AVX_H -#define _ASM_X86_XOR_AVX_H - +// SPDX-License-Identifier: GPL-2.0-only /* - * Optimized RAID-5 checksumming functions for AVX + * Optimized XOR parity functions for AVX * * Copyright (C) 2012 Intel Corporation * Author: Jim Kukunas * * Based on Ingo Molnar and Zach Brown's respective MMX and SSE routines */ - #include #include +#include "xor_impl.h" +#include "xor_arch.h" #define BLOCK4(i) \ BLOCK(32 * i, 0) \ @@ -31,8 +29,6 @@ static void xor_avx_2(unsigned long bytes, unsigned long * __restrict p0, { unsigned long lines = bytes >> 9; - kernel_fpu_begin(); - while (lines--) { #undef BLOCK #define BLOCK(i, reg) \ @@ -49,8 +45,6 @@ do { \ p0 = (unsigned long *)((uintptr_t)p0 + 512); p1 = (unsigned long *)((uintptr_t)p1 + 512); } - - kernel_fpu_end(); } static void xor_avx_3(unsigned long bytes, unsigned long * __restrict p0, @@ -59,8 +53,6 @@ static void xor_avx_3(unsigned long bytes, unsigned long * __restrict p0, { unsigned long lines = bytes >> 9; - kernel_fpu_begin(); - while (lines--) { #undef BLOCK #define BLOCK(i, reg) \ @@ -80,8 +72,6 @@ do { \ p1 = (unsigned long *)((uintptr_t)p1 + 512); p2 = (unsigned long *)((uintptr_t)p2 + 512); } - - kernel_fpu_end(); } static void xor_avx_4(unsigned long bytes, unsigned long * __restrict p0, @@ -91,8 +81,6 @@ static void xor_avx_4(unsigned long bytes, unsigned long * __restrict p0, { unsigned long lines = bytes >> 9; - kernel_fpu_begin(); - while (lines--) { #undef BLOCK #define BLOCK(i, reg) \ @@ -115,8 +103,6 @@ do { \ p2 = (unsigned long *)((uintptr_t)p2 + 512); p3 = (unsigned long *)((uintptr_t)p3 + 512); } - - kernel_fpu_end(); } static void xor_avx_5(unsigned long bytes, unsigned long * __restrict p0, @@ -127,8 +113,6 @@ static void xor_avx_5(unsigned long bytes, unsigned long * __restrict p0, { unsigned long lines = bytes >> 9; - kernel_fpu_begin(); - while (lines--) { #undef BLOCK #define BLOCK(i, reg) \ @@ -154,25 +138,19 @@ do { \ p3 = (unsigned long *)((uintptr_t)p3 + 512); p4 = (unsigned long *)((uintptr_t)p4 + 512); } +} +DO_XOR_BLOCKS(avx_inner, xor_avx_2, xor_avx_3, xor_avx_4, xor_avx_5); + +static void xor_gen_avx(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + kernel_fpu_begin(); + xor_gen_avx_inner(dest, srcs, src_cnt, bytes); kernel_fpu_end(); } -static struct xor_block_template xor_block_avx = { - .name = "avx", - .do_2 = xor_avx_2, - .do_3 = xor_avx_3, - .do_4 = xor_avx_4, - .do_5 = xor_avx_5, +struct xor_block_template xor_block_avx = { + .name = "avx", + .xor_gen = xor_gen_avx, }; - -#define AVX_XOR_SPEED \ -do { \ - if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_OSXSAVE)) \ - xor_speed(&xor_block_avx); \ -} while (0) - -#define AVX_SELECT(FASTEST) \ - (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_OSXSAVE) ? &xor_block_avx : FASTEST) - -#endif diff --git a/arch/x86/include/asm/xor_32.h b/lib/raid/xor/x86/xor-mmx.c similarity index 87% rename from arch/x86/include/asm/xor_32.h rename to lib/raid/xor/x86/xor-mmx.c index 7a6b9474591e..63a8b0444fce 100644 --- a/arch/x86/include/asm/xor_32.h +++ b/lib/raid/xor/x86/xor-mmx.c @@ -1,15 +1,12 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -#ifndef _ASM_X86_XOR_32_H -#define _ASM_X86_XOR_32_H - +// SPDX-License-Identifier: GPL-2.0-or-later /* - * Optimized RAID-5 checksumming functions for MMX. - */ - -/* - * High-speed RAID5 checksumming functions utilizing MMX instructions. + * Optimized XOR parity functions for MMX. + * * Copyright (C) 1998 Ingo Molnar. */ +#include +#include "xor_impl.h" +#include "xor_arch.h" #define LD(x, y) " movq 8*("#x")(%1), %%mm"#y" ;\n" #define ST(x, y) " movq %%mm"#y", 8*("#x")(%1) ;\n" @@ -18,16 +15,12 @@ #define XO3(x, y) " pxor 8*("#x")(%4), %%mm"#y" ;\n" #define XO4(x, y) " pxor 8*("#x")(%5), %%mm"#y" ;\n" -#include - static void xor_pII_mmx_2(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2) { unsigned long lines = bytes >> 7; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -60,8 +53,6 @@ xor_pII_mmx_2(unsigned long bytes, unsigned long * __restrict p1, "+r" (p1), "+r" (p2) : : "memory"); - - kernel_fpu_end(); } static void @@ -71,8 +62,6 @@ xor_pII_mmx_3(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 7; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -110,8 +99,6 @@ xor_pII_mmx_3(unsigned long bytes, unsigned long * __restrict p1, "+r" (p1), "+r" (p2), "+r" (p3) : : "memory"); - - kernel_fpu_end(); } static void @@ -122,8 +109,6 @@ xor_pII_mmx_4(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 7; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -166,8 +151,6 @@ xor_pII_mmx_4(unsigned long bytes, unsigned long * __restrict p1, "+r" (p1), "+r" (p2), "+r" (p3), "+r" (p4) : : "memory"); - - kernel_fpu_end(); } @@ -180,8 +163,6 @@ xor_pII_mmx_5(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 7; - kernel_fpu_begin(); - /* Make sure GCC forgets anything it knows about p4 or p5, such that it won't pass to the asm volatile below a register that is shared with any other variable. That's @@ -242,8 +223,6 @@ xor_pII_mmx_5(unsigned long bytes, unsigned long * __restrict p1, Clobber them just to be sure nobody does something stupid like assuming they have some legal value. */ asm("" : "=r" (p4), "=r" (p5)); - - kernel_fpu_end(); } #undef LD @@ -260,8 +239,6 @@ xor_p5_mmx_2(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 6; - kernel_fpu_begin(); - asm volatile( " .align 32 ;\n" " 1: ;\n" @@ -298,8 +275,6 @@ xor_p5_mmx_2(unsigned long bytes, unsigned long * __restrict p1, "+r" (p1), "+r" (p2) : : "memory"); - - kernel_fpu_end(); } static void @@ -309,8 +284,6 @@ xor_p5_mmx_3(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 6; - kernel_fpu_begin(); - asm volatile( " .align 32,0x90 ;\n" " 1: ;\n" @@ -356,8 +329,6 @@ xor_p5_mmx_3(unsigned long bytes, unsigned long * __restrict p1, "+r" (p1), "+r" (p2), "+r" (p3) : : "memory" ); - - kernel_fpu_end(); } static void @@ -368,8 +339,6 @@ xor_p5_mmx_4(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 6; - kernel_fpu_begin(); - asm volatile( " .align 32,0x90 ;\n" " 1: ;\n" @@ -424,8 +393,6 @@ xor_p5_mmx_4(unsigned long bytes, unsigned long * __restrict p1, "+r" (p1), "+r" (p2), "+r" (p3), "+r" (p4) : : "memory"); - - kernel_fpu_end(); } static void @@ -437,8 +404,6 @@ xor_p5_mmx_5(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 6; - kernel_fpu_begin(); - /* Make sure GCC forgets anything it knows about p4 or p5, such that it won't pass to the asm volatile below a register that is shared with any other variable. That's @@ -515,59 +480,36 @@ xor_p5_mmx_5(unsigned long bytes, unsigned long * __restrict p1, Clobber them just to be sure nobody does something stupid like assuming they have some legal value. */ asm("" : "=r" (p4), "=r" (p5)); +} +DO_XOR_BLOCKS(pII_mmx_inner, xor_pII_mmx_2, xor_pII_mmx_3, xor_pII_mmx_4, + xor_pII_mmx_5); + +static void xor_gen_pII_mmx(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + kernel_fpu_begin(); + xor_gen_pII_mmx_inner(dest, srcs, src_cnt, bytes); kernel_fpu_end(); } -static struct xor_block_template xor_block_pII_mmx = { - .name = "pII_mmx", - .do_2 = xor_pII_mmx_2, - .do_3 = xor_pII_mmx_3, - .do_4 = xor_pII_mmx_4, - .do_5 = xor_pII_mmx_5, +struct xor_block_template xor_block_pII_mmx = { + .name = "pII_mmx", + .xor_gen = xor_gen_pII_mmx, }; -static struct xor_block_template xor_block_p5_mmx = { - .name = "p5_mmx", - .do_2 = xor_p5_mmx_2, - .do_3 = xor_p5_mmx_3, - .do_4 = xor_p5_mmx_4, - .do_5 = xor_p5_mmx_5, +DO_XOR_BLOCKS(p5_mmx_inner, xor_p5_mmx_2, xor_p5_mmx_3, xor_p5_mmx_4, + xor_p5_mmx_5); + +static void xor_gen_p5_mmx(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + kernel_fpu_begin(); + xor_gen_p5_mmx_inner(dest, srcs, src_cnt, bytes); + kernel_fpu_end(); +} + +struct xor_block_template xor_block_p5_mmx = { + .name = "p5_mmx", + .xor_gen = xor_gen_p5_mmx, }; - -static struct xor_block_template xor_block_pIII_sse = { - .name = "pIII_sse", - .do_2 = xor_sse_2, - .do_3 = xor_sse_3, - .do_4 = xor_sse_4, - .do_5 = xor_sse_5, -}; - -/* Also try the AVX routines */ -#include - -/* Also try the generic routines. */ -#include - -/* We force the use of the SSE xor block because it can write around L2. - We may also be able to load into the L1 only depending on how the cpu - deals with a load to a line that is being prefetched. */ -#undef XOR_TRY_TEMPLATES -#define XOR_TRY_TEMPLATES \ -do { \ - AVX_XOR_SPEED; \ - if (boot_cpu_has(X86_FEATURE_XMM)) { \ - xor_speed(&xor_block_pIII_sse); \ - xor_speed(&xor_block_sse_pf64); \ - } else if (boot_cpu_has(X86_FEATURE_MMX)) { \ - xor_speed(&xor_block_pII_mmx); \ - xor_speed(&xor_block_p5_mmx); \ - } else { \ - xor_speed(&xor_block_8regs); \ - xor_speed(&xor_block_8regs_p); \ - xor_speed(&xor_block_32regs); \ - xor_speed(&xor_block_32regs_p); \ - } \ -} while (0) - -#endif /* _ASM_X86_XOR_32_H */ diff --git a/arch/x86/include/asm/xor.h b/lib/raid/xor/x86/xor-sse.c similarity index 90% rename from arch/x86/include/asm/xor.h rename to lib/raid/xor/x86/xor-sse.c index 7b0307acc410..c6626ecae6ba 100644 --- a/arch/x86/include/asm/xor.h +++ b/lib/raid/xor/x86/xor-sse.c @@ -1,31 +1,20 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -#ifndef _ASM_X86_XOR_H -#define _ASM_X86_XOR_H - -/* - * Optimized RAID-5 checksumming functions for SSE. - */ - +// SPDX-License-Identifier: GPL-2.0-or-later /* + * Optimized XOR parity functions for SSE. + * * Cache avoiding checksumming functions utilizing KNI instructions * Copyright (C) 1999 Zach Brown (with obvious credit due Ingo) - */ - -/* + * * Based on * High-speed RAID5 checksumming functions utilizing SSE instructions. * Copyright (C) 1998 Ingo Molnar. - */ - -/* + * * x86-64 changes / gcc fixes from Andi Kleen. * Copyright 2002 Andi Kleen, SuSE Labs. - * - * This hasn't been optimized for the hammer yet, but there are likely - * no advantages to be gotten from x86-64 here anyways. */ - #include +#include "xor_impl.h" +#include "xor_arch.h" #ifdef CONFIG_X86_32 /* reduce register pressure */ @@ -62,8 +51,6 @@ xor_sse_2(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 8; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -104,8 +91,6 @@ xor_sse_2(unsigned long bytes, unsigned long * __restrict p1, [p1] "+r" (p1), [p2] "+r" (p2) : [inc] XOR_CONSTANT_CONSTRAINT (256UL) : "memory"); - - kernel_fpu_end(); } static void @@ -114,8 +99,6 @@ xor_sse_2_pf64(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 8; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -139,8 +122,6 @@ xor_sse_2_pf64(unsigned long bytes, unsigned long * __restrict p1, [p1] "+r" (p1), [p2] "+r" (p2) : [inc] XOR_CONSTANT_CONSTRAINT (256UL) : "memory"); - - kernel_fpu_end(); } static void @@ -150,8 +131,6 @@ xor_sse_3(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 8; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -199,8 +178,6 @@ xor_sse_3(unsigned long bytes, unsigned long * __restrict p1, [p1] "+r" (p1), [p2] "+r" (p2), [p3] "+r" (p3) : [inc] XOR_CONSTANT_CONSTRAINT (256UL) : "memory"); - - kernel_fpu_end(); } static void @@ -210,8 +187,6 @@ xor_sse_3_pf64(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 8; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -237,8 +212,6 @@ xor_sse_3_pf64(unsigned long bytes, unsigned long * __restrict p1, [p1] "+r" (p1), [p2] "+r" (p2), [p3] "+r" (p3) : [inc] XOR_CONSTANT_CONSTRAINT (256UL) : "memory"); - - kernel_fpu_end(); } static void @@ -249,8 +222,6 @@ xor_sse_4(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 8; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -305,8 +276,6 @@ xor_sse_4(unsigned long bytes, unsigned long * __restrict p1, [p2] "+r" (p2), [p3] "+r" (p3), [p4] "+r" (p4) : [inc] XOR_CONSTANT_CONSTRAINT (256UL) : "memory"); - - kernel_fpu_end(); } static void @@ -317,8 +286,6 @@ xor_sse_4_pf64(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 8; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -346,8 +313,6 @@ xor_sse_4_pf64(unsigned long bytes, unsigned long * __restrict p1, [p2] "+r" (p2), [p3] "+r" (p3), [p4] "+r" (p4) : [inc] XOR_CONSTANT_CONSTRAINT (256UL) : "memory"); - - kernel_fpu_end(); } static void @@ -359,8 +324,6 @@ xor_sse_5(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 8; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -422,8 +385,6 @@ xor_sse_5(unsigned long bytes, unsigned long * __restrict p1, [p3] "+r" (p3), [p4] "+r" (p4), [p5] "+r" (p5) : [inc] XOR_CONSTANT_CONSTRAINT (256UL) : "memory"); - - kernel_fpu_end(); } static void @@ -435,8 +396,6 @@ xor_sse_5_pf64(unsigned long bytes, unsigned long * __restrict p1, { unsigned long lines = bytes >> 8; - kernel_fpu_begin(); - asm volatile( #undef BLOCK #define BLOCK(i) \ @@ -466,37 +425,35 @@ xor_sse_5_pf64(unsigned long bytes, unsigned long * __restrict p1, [p3] "+r" (p3), [p4] "+r" (p4), [p5] "+r" (p5) : [inc] XOR_CONSTANT_CONSTRAINT (256UL) : "memory"); +} +DO_XOR_BLOCKS(sse_inner, xor_sse_2, xor_sse_3, xor_sse_4, xor_sse_5); + +static void xor_gen_sse(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + kernel_fpu_begin(); + xor_gen_sse_inner(dest, srcs, src_cnt, bytes); kernel_fpu_end(); } -static struct xor_block_template xor_block_sse_pf64 = { - .name = "prefetch64-sse", - .do_2 = xor_sse_2_pf64, - .do_3 = xor_sse_3_pf64, - .do_4 = xor_sse_4_pf64, - .do_5 = xor_sse_5_pf64, +struct xor_block_template xor_block_sse = { + .name = "sse", + .xor_gen = xor_gen_sse, }; -#undef LD -#undef XO1 -#undef XO2 -#undef XO3 -#undef XO4 -#undef ST -#undef NOP -#undef BLK64 -#undef BLOCK +DO_XOR_BLOCKS(sse_pf64_inner, xor_sse_2_pf64, xor_sse_3_pf64, xor_sse_4_pf64, + xor_sse_5_pf64); -#undef XOR_CONSTANT_CONSTRAINT +static void xor_gen_sse_pf64(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + kernel_fpu_begin(); + xor_gen_sse_pf64_inner(dest, srcs, src_cnt, bytes); + kernel_fpu_end(); +} -#ifdef CONFIG_X86_32 -# include -#else -# include -#endif - -#define XOR_SELECT_TEMPLATE(FASTEST) \ - AVX_SELECT(FASTEST) - -#endif /* _ASM_X86_XOR_H */ +struct xor_block_template xor_block_sse_pf64 = { + .name = "prefetch64-sse", + .xor_gen = xor_gen_sse_pf64, +}; diff --git a/lib/raid/xor/x86/xor_arch.h b/lib/raid/xor/x86/xor_arch.h new file mode 100644 index 000000000000..99fe85a213c6 --- /dev/null +++ b/lib/raid/xor/x86/xor_arch.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#include + +extern struct xor_block_template xor_block_pII_mmx; +extern struct xor_block_template xor_block_p5_mmx; +extern struct xor_block_template xor_block_sse; +extern struct xor_block_template xor_block_sse_pf64; +extern struct xor_block_template xor_block_avx; + +/* + * When SSE is available, use it as it can write around L2. We may also be able + * to load into the L1 only depending on how the cpu deals with a load to a line + * that is being prefetched. + * + * When AVX2 is available, force using it as it is better by all measures. + * + * 32-bit without MMX can fall back to the generic routines. + */ +static __always_inline void __init arch_xor_init(void) +{ + if (boot_cpu_has(X86_FEATURE_AVX) && + boot_cpu_has(X86_FEATURE_OSXSAVE)) { + xor_force(&xor_block_avx); + } else if (IS_ENABLED(CONFIG_X86_64) || boot_cpu_has(X86_FEATURE_XMM)) { + xor_register(&xor_block_sse); + xor_register(&xor_block_sse_pf64); + } else if (boot_cpu_has(X86_FEATURE_MMX)) { + xor_register(&xor_block_pII_mmx); + xor_register(&xor_block_p5_mmx); + } else { + xor_register(&xor_block_8regs); + xor_register(&xor_block_8regs_p); + xor_register(&xor_block_32regs); + xor_register(&xor_block_32regs_p); + } +} diff --git a/lib/raid/xor/xor-32regs-prefetch.c b/lib/raid/xor/xor-32regs-prefetch.c new file mode 100644 index 000000000000..ade2a7d8cbe2 --- /dev/null +++ b/lib/raid/xor/xor-32regs-prefetch.c @@ -0,0 +1,267 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include "xor_impl.h" + +static void +xor_32regs_p_2(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2) +{ + long lines = bytes / (sizeof (long)) / 8 - 1; + + prefetchw(p1); + prefetch(p2); + + do { + register long d0, d1, d2, d3, d4, d5, d6, d7; + + prefetchw(p1+8); + prefetch(p2+8); + once_more: + d0 = p1[0]; /* Pull the stuff into registers */ + d1 = p1[1]; /* ... in bursts, if possible. */ + d2 = p1[2]; + d3 = p1[3]; + d4 = p1[4]; + d5 = p1[5]; + d6 = p1[6]; + d7 = p1[7]; + d0 ^= p2[0]; + d1 ^= p2[1]; + d2 ^= p2[2]; + d3 ^= p2[3]; + d4 ^= p2[4]; + d5 ^= p2[5]; + d6 ^= p2[6]; + d7 ^= p2[7]; + p1[0] = d0; /* Store the result (in bursts) */ + p1[1] = d1; + p1[2] = d2; + p1[3] = d3; + p1[4] = d4; + p1[5] = d5; + p1[6] = d6; + p1[7] = d7; + p1 += 8; + p2 += 8; + } while (--lines > 0); + if (lines == 0) + goto once_more; +} + +static void +xor_32regs_p_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3) +{ + long lines = bytes / (sizeof (long)) / 8 - 1; + + prefetchw(p1); + prefetch(p2); + prefetch(p3); + + do { + register long d0, d1, d2, d3, d4, d5, d6, d7; + + prefetchw(p1+8); + prefetch(p2+8); + prefetch(p3+8); + once_more: + d0 = p1[0]; /* Pull the stuff into registers */ + d1 = p1[1]; /* ... in bursts, if possible. */ + d2 = p1[2]; + d3 = p1[3]; + d4 = p1[4]; + d5 = p1[5]; + d6 = p1[6]; + d7 = p1[7]; + d0 ^= p2[0]; + d1 ^= p2[1]; + d2 ^= p2[2]; + d3 ^= p2[3]; + d4 ^= p2[4]; + d5 ^= p2[5]; + d6 ^= p2[6]; + d7 ^= p2[7]; + d0 ^= p3[0]; + d1 ^= p3[1]; + d2 ^= p3[2]; + d3 ^= p3[3]; + d4 ^= p3[4]; + d5 ^= p3[5]; + d6 ^= p3[6]; + d7 ^= p3[7]; + p1[0] = d0; /* Store the result (in bursts) */ + p1[1] = d1; + p1[2] = d2; + p1[3] = d3; + p1[4] = d4; + p1[5] = d5; + p1[6] = d6; + p1[7] = d7; + p1 += 8; + p2 += 8; + p3 += 8; + } while (--lines > 0); + if (lines == 0) + goto once_more; +} + +static void +xor_32regs_p_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4) +{ + long lines = bytes / (sizeof (long)) / 8 - 1; + + prefetchw(p1); + prefetch(p2); + prefetch(p3); + prefetch(p4); + + do { + register long d0, d1, d2, d3, d4, d5, d6, d7; + + prefetchw(p1+8); + prefetch(p2+8); + prefetch(p3+8); + prefetch(p4+8); + once_more: + d0 = p1[0]; /* Pull the stuff into registers */ + d1 = p1[1]; /* ... in bursts, if possible. */ + d2 = p1[2]; + d3 = p1[3]; + d4 = p1[4]; + d5 = p1[5]; + d6 = p1[6]; + d7 = p1[7]; + d0 ^= p2[0]; + d1 ^= p2[1]; + d2 ^= p2[2]; + d3 ^= p2[3]; + d4 ^= p2[4]; + d5 ^= p2[5]; + d6 ^= p2[6]; + d7 ^= p2[7]; + d0 ^= p3[0]; + d1 ^= p3[1]; + d2 ^= p3[2]; + d3 ^= p3[3]; + d4 ^= p3[4]; + d5 ^= p3[5]; + d6 ^= p3[6]; + d7 ^= p3[7]; + d0 ^= p4[0]; + d1 ^= p4[1]; + d2 ^= p4[2]; + d3 ^= p4[3]; + d4 ^= p4[4]; + d5 ^= p4[5]; + d6 ^= p4[6]; + d7 ^= p4[7]; + p1[0] = d0; /* Store the result (in bursts) */ + p1[1] = d1; + p1[2] = d2; + p1[3] = d3; + p1[4] = d4; + p1[5] = d5; + p1[6] = d6; + p1[7] = d7; + p1 += 8; + p2 += 8; + p3 += 8; + p4 += 8; + } while (--lines > 0); + if (lines == 0) + goto once_more; +} + +static void +xor_32regs_p_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4, + const unsigned long * __restrict p5) +{ + long lines = bytes / (sizeof (long)) / 8 - 1; + + prefetchw(p1); + prefetch(p2); + prefetch(p3); + prefetch(p4); + prefetch(p5); + + do { + register long d0, d1, d2, d3, d4, d5, d6, d7; + + prefetchw(p1+8); + prefetch(p2+8); + prefetch(p3+8); + prefetch(p4+8); + prefetch(p5+8); + once_more: + d0 = p1[0]; /* Pull the stuff into registers */ + d1 = p1[1]; /* ... in bursts, if possible. */ + d2 = p1[2]; + d3 = p1[3]; + d4 = p1[4]; + d5 = p1[5]; + d6 = p1[6]; + d7 = p1[7]; + d0 ^= p2[0]; + d1 ^= p2[1]; + d2 ^= p2[2]; + d3 ^= p2[3]; + d4 ^= p2[4]; + d5 ^= p2[5]; + d6 ^= p2[6]; + d7 ^= p2[7]; + d0 ^= p3[0]; + d1 ^= p3[1]; + d2 ^= p3[2]; + d3 ^= p3[3]; + d4 ^= p3[4]; + d5 ^= p3[5]; + d6 ^= p3[6]; + d7 ^= p3[7]; + d0 ^= p4[0]; + d1 ^= p4[1]; + d2 ^= p4[2]; + d3 ^= p4[3]; + d4 ^= p4[4]; + d5 ^= p4[5]; + d6 ^= p4[6]; + d7 ^= p4[7]; + d0 ^= p5[0]; + d1 ^= p5[1]; + d2 ^= p5[2]; + d3 ^= p5[3]; + d4 ^= p5[4]; + d5 ^= p5[5]; + d6 ^= p5[6]; + d7 ^= p5[7]; + p1[0] = d0; /* Store the result (in bursts) */ + p1[1] = d1; + p1[2] = d2; + p1[3] = d3; + p1[4] = d4; + p1[5] = d5; + p1[6] = d6; + p1[7] = d7; + p1 += 8; + p2 += 8; + p3 += 8; + p4 += 8; + p5 += 8; + } while (--lines > 0); + if (lines == 0) + goto once_more; +} + +DO_XOR_BLOCKS(32regs_p, xor_32regs_p_2, xor_32regs_p_3, xor_32regs_p_4, + xor_32regs_p_5); + +struct xor_block_template xor_block_32regs_p = { + .name = "32regs_prefetch", + .xor_gen = xor_gen_32regs_p, +}; diff --git a/lib/raid/xor/xor-32regs.c b/lib/raid/xor/xor-32regs.c new file mode 100644 index 000000000000..acb4a10d1e95 --- /dev/null +++ b/lib/raid/xor/xor-32regs.c @@ -0,0 +1,217 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include "xor_impl.h" + +static void +xor_32regs_2(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2) +{ + long lines = bytes / (sizeof (long)) / 8; + + do { + register long d0, d1, d2, d3, d4, d5, d6, d7; + d0 = p1[0]; /* Pull the stuff into registers */ + d1 = p1[1]; /* ... in bursts, if possible. */ + d2 = p1[2]; + d3 = p1[3]; + d4 = p1[4]; + d5 = p1[5]; + d6 = p1[6]; + d7 = p1[7]; + d0 ^= p2[0]; + d1 ^= p2[1]; + d2 ^= p2[2]; + d3 ^= p2[3]; + d4 ^= p2[4]; + d5 ^= p2[5]; + d6 ^= p2[6]; + d7 ^= p2[7]; + p1[0] = d0; /* Store the result (in bursts) */ + p1[1] = d1; + p1[2] = d2; + p1[3] = d3; + p1[4] = d4; + p1[5] = d5; + p1[6] = d6; + p1[7] = d7; + p1 += 8; + p2 += 8; + } while (--lines > 0); +} + +static void +xor_32regs_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3) +{ + long lines = bytes / (sizeof (long)) / 8; + + do { + register long d0, d1, d2, d3, d4, d5, d6, d7; + d0 = p1[0]; /* Pull the stuff into registers */ + d1 = p1[1]; /* ... in bursts, if possible. */ + d2 = p1[2]; + d3 = p1[3]; + d4 = p1[4]; + d5 = p1[5]; + d6 = p1[6]; + d7 = p1[7]; + d0 ^= p2[0]; + d1 ^= p2[1]; + d2 ^= p2[2]; + d3 ^= p2[3]; + d4 ^= p2[4]; + d5 ^= p2[5]; + d6 ^= p2[6]; + d7 ^= p2[7]; + d0 ^= p3[0]; + d1 ^= p3[1]; + d2 ^= p3[2]; + d3 ^= p3[3]; + d4 ^= p3[4]; + d5 ^= p3[5]; + d6 ^= p3[6]; + d7 ^= p3[7]; + p1[0] = d0; /* Store the result (in bursts) */ + p1[1] = d1; + p1[2] = d2; + p1[3] = d3; + p1[4] = d4; + p1[5] = d5; + p1[6] = d6; + p1[7] = d7; + p1 += 8; + p2 += 8; + p3 += 8; + } while (--lines > 0); +} + +static void +xor_32regs_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4) +{ + long lines = bytes / (sizeof (long)) / 8; + + do { + register long d0, d1, d2, d3, d4, d5, d6, d7; + d0 = p1[0]; /* Pull the stuff into registers */ + d1 = p1[1]; /* ... in bursts, if possible. */ + d2 = p1[2]; + d3 = p1[3]; + d4 = p1[4]; + d5 = p1[5]; + d6 = p1[6]; + d7 = p1[7]; + d0 ^= p2[0]; + d1 ^= p2[1]; + d2 ^= p2[2]; + d3 ^= p2[3]; + d4 ^= p2[4]; + d5 ^= p2[5]; + d6 ^= p2[6]; + d7 ^= p2[7]; + d0 ^= p3[0]; + d1 ^= p3[1]; + d2 ^= p3[2]; + d3 ^= p3[3]; + d4 ^= p3[4]; + d5 ^= p3[5]; + d6 ^= p3[6]; + d7 ^= p3[7]; + d0 ^= p4[0]; + d1 ^= p4[1]; + d2 ^= p4[2]; + d3 ^= p4[3]; + d4 ^= p4[4]; + d5 ^= p4[5]; + d6 ^= p4[6]; + d7 ^= p4[7]; + p1[0] = d0; /* Store the result (in bursts) */ + p1[1] = d1; + p1[2] = d2; + p1[3] = d3; + p1[4] = d4; + p1[5] = d5; + p1[6] = d6; + p1[7] = d7; + p1 += 8; + p2 += 8; + p3 += 8; + p4 += 8; + } while (--lines > 0); +} + +static void +xor_32regs_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4, + const unsigned long * __restrict p5) +{ + long lines = bytes / (sizeof (long)) / 8; + + do { + register long d0, d1, d2, d3, d4, d5, d6, d7; + d0 = p1[0]; /* Pull the stuff into registers */ + d1 = p1[1]; /* ... in bursts, if possible. */ + d2 = p1[2]; + d3 = p1[3]; + d4 = p1[4]; + d5 = p1[5]; + d6 = p1[6]; + d7 = p1[7]; + d0 ^= p2[0]; + d1 ^= p2[1]; + d2 ^= p2[2]; + d3 ^= p2[3]; + d4 ^= p2[4]; + d5 ^= p2[5]; + d6 ^= p2[6]; + d7 ^= p2[7]; + d0 ^= p3[0]; + d1 ^= p3[1]; + d2 ^= p3[2]; + d3 ^= p3[3]; + d4 ^= p3[4]; + d5 ^= p3[5]; + d6 ^= p3[6]; + d7 ^= p3[7]; + d0 ^= p4[0]; + d1 ^= p4[1]; + d2 ^= p4[2]; + d3 ^= p4[3]; + d4 ^= p4[4]; + d5 ^= p4[5]; + d6 ^= p4[6]; + d7 ^= p4[7]; + d0 ^= p5[0]; + d1 ^= p5[1]; + d2 ^= p5[2]; + d3 ^= p5[3]; + d4 ^= p5[4]; + d5 ^= p5[5]; + d6 ^= p5[6]; + d7 ^= p5[7]; + p1[0] = d0; /* Store the result (in bursts) */ + p1[1] = d1; + p1[2] = d2; + p1[3] = d3; + p1[4] = d4; + p1[5] = d5; + p1[6] = d6; + p1[7] = d7; + p1 += 8; + p2 += 8; + p3 += 8; + p4 += 8; + p5 += 8; + } while (--lines > 0); +} + +DO_XOR_BLOCKS(32regs, xor_32regs_2, xor_32regs_3, xor_32regs_4, xor_32regs_5); + +struct xor_block_template xor_block_32regs = { + .name = "32regs", + .xor_gen = xor_gen_32regs, +}; diff --git a/lib/raid/xor/xor-8regs-prefetch.c b/lib/raid/xor/xor-8regs-prefetch.c new file mode 100644 index 000000000000..451527a951b1 --- /dev/null +++ b/lib/raid/xor/xor-8regs-prefetch.c @@ -0,0 +1,146 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include "xor_impl.h" + +static void +xor_8regs_p_2(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2) +{ + long lines = bytes / (sizeof (long)) / 8 - 1; + prefetchw(p1); + prefetch(p2); + + do { + prefetchw(p1+8); + prefetch(p2+8); + once_more: + p1[0] ^= p2[0]; + p1[1] ^= p2[1]; + p1[2] ^= p2[2]; + p1[3] ^= p2[3]; + p1[4] ^= p2[4]; + p1[5] ^= p2[5]; + p1[6] ^= p2[6]; + p1[7] ^= p2[7]; + p1 += 8; + p2 += 8; + } while (--lines > 0); + if (lines == 0) + goto once_more; +} + +static void +xor_8regs_p_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3) +{ + long lines = bytes / (sizeof (long)) / 8 - 1; + prefetchw(p1); + prefetch(p2); + prefetch(p3); + + do { + prefetchw(p1+8); + prefetch(p2+8); + prefetch(p3+8); + once_more: + p1[0] ^= p2[0] ^ p3[0]; + p1[1] ^= p2[1] ^ p3[1]; + p1[2] ^= p2[2] ^ p3[2]; + p1[3] ^= p2[3] ^ p3[3]; + p1[4] ^= p2[4] ^ p3[4]; + p1[5] ^= p2[5] ^ p3[5]; + p1[6] ^= p2[6] ^ p3[6]; + p1[7] ^= p2[7] ^ p3[7]; + p1 += 8; + p2 += 8; + p3 += 8; + } while (--lines > 0); + if (lines == 0) + goto once_more; +} + +static void +xor_8regs_p_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4) +{ + long lines = bytes / (sizeof (long)) / 8 - 1; + + prefetchw(p1); + prefetch(p2); + prefetch(p3); + prefetch(p4); + + do { + prefetchw(p1+8); + prefetch(p2+8); + prefetch(p3+8); + prefetch(p4+8); + once_more: + p1[0] ^= p2[0] ^ p3[0] ^ p4[0]; + p1[1] ^= p2[1] ^ p3[1] ^ p4[1]; + p1[2] ^= p2[2] ^ p3[2] ^ p4[2]; + p1[3] ^= p2[3] ^ p3[3] ^ p4[3]; + p1[4] ^= p2[4] ^ p3[4] ^ p4[4]; + p1[5] ^= p2[5] ^ p3[5] ^ p4[5]; + p1[6] ^= p2[6] ^ p3[6] ^ p4[6]; + p1[7] ^= p2[7] ^ p3[7] ^ p4[7]; + p1 += 8; + p2 += 8; + p3 += 8; + p4 += 8; + } while (--lines > 0); + if (lines == 0) + goto once_more; +} + +static void +xor_8regs_p_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4, + const unsigned long * __restrict p5) +{ + long lines = bytes / (sizeof (long)) / 8 - 1; + + prefetchw(p1); + prefetch(p2); + prefetch(p3); + prefetch(p4); + prefetch(p5); + + do { + prefetchw(p1+8); + prefetch(p2+8); + prefetch(p3+8); + prefetch(p4+8); + prefetch(p5+8); + once_more: + p1[0] ^= p2[0] ^ p3[0] ^ p4[0] ^ p5[0]; + p1[1] ^= p2[1] ^ p3[1] ^ p4[1] ^ p5[1]; + p1[2] ^= p2[2] ^ p3[2] ^ p4[2] ^ p5[2]; + p1[3] ^= p2[3] ^ p3[3] ^ p4[3] ^ p5[3]; + p1[4] ^= p2[4] ^ p3[4] ^ p4[4] ^ p5[4]; + p1[5] ^= p2[5] ^ p3[5] ^ p4[5] ^ p5[5]; + p1[6] ^= p2[6] ^ p3[6] ^ p4[6] ^ p5[6]; + p1[7] ^= p2[7] ^ p3[7] ^ p4[7] ^ p5[7]; + p1 += 8; + p2 += 8; + p3 += 8; + p4 += 8; + p5 += 8; + } while (--lines > 0); + if (lines == 0) + goto once_more; +} + + +DO_XOR_BLOCKS(8regs_p, xor_8regs_p_2, xor_8regs_p_3, xor_8regs_p_4, + xor_8regs_p_5); + +struct xor_block_template xor_block_8regs_p = { + .name = "8regs_prefetch", + .xor_gen = xor_gen_8regs_p, +}; diff --git a/lib/raid/xor/xor-8regs.c b/lib/raid/xor/xor-8regs.c new file mode 100644 index 000000000000..1edaed8acffe --- /dev/null +++ b/lib/raid/xor/xor-8regs.c @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include "xor_impl.h" + +static void +xor_8regs_2(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2) +{ + long lines = bytes / (sizeof (long)) / 8; + + do { + p1[0] ^= p2[0]; + p1[1] ^= p2[1]; + p1[2] ^= p2[2]; + p1[3] ^= p2[3]; + p1[4] ^= p2[4]; + p1[5] ^= p2[5]; + p1[6] ^= p2[6]; + p1[7] ^= p2[7]; + p1 += 8; + p2 += 8; + } while (--lines > 0); +} + +static void +xor_8regs_3(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3) +{ + long lines = bytes / (sizeof (long)) / 8; + + do { + p1[0] ^= p2[0] ^ p3[0]; + p1[1] ^= p2[1] ^ p3[1]; + p1[2] ^= p2[2] ^ p3[2]; + p1[3] ^= p2[3] ^ p3[3]; + p1[4] ^= p2[4] ^ p3[4]; + p1[5] ^= p2[5] ^ p3[5]; + p1[6] ^= p2[6] ^ p3[6]; + p1[7] ^= p2[7] ^ p3[7]; + p1 += 8; + p2 += 8; + p3 += 8; + } while (--lines > 0); +} + +static void +xor_8regs_4(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4) +{ + long lines = bytes / (sizeof (long)) / 8; + + do { + p1[0] ^= p2[0] ^ p3[0] ^ p4[0]; + p1[1] ^= p2[1] ^ p3[1] ^ p4[1]; + p1[2] ^= p2[2] ^ p3[2] ^ p4[2]; + p1[3] ^= p2[3] ^ p3[3] ^ p4[3]; + p1[4] ^= p2[4] ^ p3[4] ^ p4[4]; + p1[5] ^= p2[5] ^ p3[5] ^ p4[5]; + p1[6] ^= p2[6] ^ p3[6] ^ p4[6]; + p1[7] ^= p2[7] ^ p3[7] ^ p4[7]; + p1 += 8; + p2 += 8; + p3 += 8; + p4 += 8; + } while (--lines > 0); +} + +static void +xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1, + const unsigned long * __restrict p2, + const unsigned long * __restrict p3, + const unsigned long * __restrict p4, + const unsigned long * __restrict p5) +{ + long lines = bytes / (sizeof (long)) / 8; + + do { + p1[0] ^= p2[0] ^ p3[0] ^ p4[0] ^ p5[0]; + p1[1] ^= p2[1] ^ p3[1] ^ p4[1] ^ p5[1]; + p1[2] ^= p2[2] ^ p3[2] ^ p4[2] ^ p5[2]; + p1[3] ^= p2[3] ^ p3[3] ^ p4[3] ^ p5[3]; + p1[4] ^= p2[4] ^ p3[4] ^ p4[4] ^ p5[4]; + p1[5] ^= p2[5] ^ p3[5] ^ p4[5] ^ p5[5]; + p1[6] ^= p2[6] ^ p3[6] ^ p4[6] ^ p5[6]; + p1[7] ^= p2[7] ^ p3[7] ^ p4[7] ^ p5[7]; + p1 += 8; + p2 += 8; + p3 += 8; + p4 += 8; + p5 += 8; + } while (--lines > 0); +} + +#ifndef NO_TEMPLATE +DO_XOR_BLOCKS(8regs, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5); + +struct xor_block_template xor_block_8regs = { + .name = "8regs", + .xor_gen = xor_gen_8regs, +}; +#endif /* NO_TEMPLATE */ diff --git a/lib/raid/xor/xor-core.c b/lib/raid/xor/xor-core.c new file mode 100644 index 000000000000..bd4e6e434418 --- /dev/null +++ b/lib/raid/xor/xor-core.c @@ -0,0 +1,193 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 1996, 1997, 1998, 1999, 2000, + * Ingo Molnar, Matti Aarnio, Jakub Jelinek, Richard Henderson. + * + * Dispatch optimized XOR parity functions. + */ + +#include +#include +#include +#include +#include +#include +#include "xor_impl.h" + +DEFINE_STATIC_CALL_NULL(xor_gen_impl, *xor_block_8regs.xor_gen); + +/** + * xor_gen - generate RAID-style XOR information + * @dest: destination vector + * @srcs: source vectors + * @src_cnt: number of source vectors + * @bytes: length in bytes of each vector + * + * Performs bit-wise XOR operation into @dest for each of the @src_cnt vectors + * in @srcs for a length of @bytes bytes. @src_cnt must be non-zero, and the + * memory pointed to by @dest and each member of @srcs must be at least 64-byte + * aligned. @bytes must be non-zero and a multiple of 512. + * + * Note: for typical RAID uses, @dest either needs to be zeroed, or filled with + * the first disk, which then needs to be removed from @srcs. + */ +void xor_gen(void *dest, void **srcs, unsigned int src_cnt, unsigned int bytes) +{ + WARN_ON_ONCE(!in_task() || irqs_disabled() || softirq_count()); + WARN_ON_ONCE(bytes == 0); + WARN_ON_ONCE(bytes & 511); + + static_call(xor_gen_impl)(dest, srcs, src_cnt, bytes); +} +EXPORT_SYMBOL(xor_gen); + +/* Set of all registered templates. */ +static struct xor_block_template *__initdata template_list; +static struct xor_block_template *forced_template; + +/** + * xor_register - register a XOR template + * @tmpl: template to register + * + * Register a XOR implementation with the core. Registered implementations + * will be measured by a trivial benchmark, and the fastest one is chosen + * unless an implementation is forced using xor_force(). + */ +void __init xor_register(struct xor_block_template *tmpl) +{ + tmpl->next = template_list; + template_list = tmpl; +} + +/** + * xor_force - force use of a XOR template + * @tmpl: template to register + * + * Register a XOR implementation with the core and force using it. Forcing + * an implementation will make the core ignore any template registered using + * xor_register(), or any previous implementation forced using xor_force(). + */ +void __init xor_force(struct xor_block_template *tmpl) +{ + forced_template = tmpl; +} + +#define BENCH_SIZE 4096 +#define REPS 800U + +static void __init +do_xor_speed(struct xor_block_template *tmpl, void *b1, void *b2) +{ + int speed; + unsigned long reps; + ktime_t min, start, t0; + void *srcs[1] = { b2 }; + + preempt_disable(); + + reps = 0; + t0 = ktime_get(); + /* delay start until time has advanced */ + while ((start = ktime_get()) == t0) + cpu_relax(); + do { + mb(); /* prevent loop optimization */ + tmpl->xor_gen(b1, srcs, 1, BENCH_SIZE); + mb(); + } while (reps++ < REPS || (t0 = ktime_get()) == start); + min = ktime_sub(t0, start); + + preempt_enable(); + + // bytes/ns == GB/s, multiply by 1000 to get MB/s [not MiB/s] + speed = (1000 * reps * BENCH_SIZE) / (unsigned int)ktime_to_ns(min); + tmpl->speed = speed; + + pr_info(" %-16s: %5d MB/sec\n", tmpl->name, speed); +} + +static int __init calibrate_xor_blocks(void) +{ + void *b1, *b2; + struct xor_block_template *f, *fastest; + + if (forced_template) + return 0; + + b1 = (void *) __get_free_pages(GFP_KERNEL, 2); + if (!b1) { + pr_warn("xor: Yikes! No memory available.\n"); + return -ENOMEM; + } + b2 = b1 + 2*PAGE_SIZE + BENCH_SIZE; + + pr_info("xor: measuring software checksum speed\n"); + fastest = template_list; + for (f = template_list; f; f = f->next) { + do_xor_speed(f, b1, b2); + if (f->speed > fastest->speed) + fastest = f; + } + static_call_update(xor_gen_impl, fastest->xor_gen); + pr_info("xor: using function: %s (%d MB/sec)\n", + fastest->name, fastest->speed); + + free_pages((unsigned long)b1, 2); + return 0; +} + +#ifdef CONFIG_XOR_BLOCKS_ARCH +#include "xor_arch.h" /* $SRCARCH/xor_arch.h */ +#else +static void __init arch_xor_init(void) +{ + xor_register(&xor_block_8regs); + xor_register(&xor_block_8regs_p); + xor_register(&xor_block_32regs); + xor_register(&xor_block_32regs_p); +} +#endif /* CONFIG_XOR_BLOCKS_ARCH */ + +static int __init xor_init(void) +{ + arch_xor_init(); + + /* + * If this arch/cpu has a short-circuited selection, don't loop through + * all the possible functions, just use the best one. + */ + if (forced_template) { + pr_info("xor: automatically using best checksumming function %-10s\n", + forced_template->name); + static_call_update(xor_gen_impl, forced_template->xor_gen); + return 0; + } + +#ifdef MODULE + return calibrate_xor_blocks(); +#else + /* + * Pick the first template as the temporary default until calibration + * happens. + */ + static_call_update(xor_gen_impl, template_list->xor_gen); + return 0; +#endif +} + +static __exit void xor_exit(void) +{ +} + +MODULE_DESCRIPTION("RAID-5 checksumming functions"); +MODULE_LICENSE("GPL"); + +/* + * When built-in we must register the default template before md, but we don't + * want calibration to run that early as that would delay the boot process. + */ +#ifndef MODULE +__initcall(calibrate_xor_blocks); +#endif +core_initcall(xor_init); +module_exit(xor_exit); diff --git a/lib/raid/xor/xor_impl.h b/lib/raid/xor/xor_impl.h new file mode 100644 index 000000000000..09ae2916f71e --- /dev/null +++ b/lib/raid/xor/xor_impl.h @@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _XOR_IMPL_H +#define _XOR_IMPL_H + +#include +#include + +struct xor_block_template { + struct xor_block_template *next; + const char *name; + int speed; + void (*xor_gen)(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes); +}; + +#define __DO_XOR_BLOCKS(_name, _handle1, _handle2, _handle3, _handle4) \ +void \ +xor_gen_##_name(void *dest, void **srcs, unsigned int src_cnt, \ + unsigned int bytes) \ +{ \ + unsigned int src_off = 0; \ + \ + while (src_cnt > 0) { \ + unsigned int this_cnt = min(src_cnt, 4); \ + \ + if (this_cnt == 1) \ + _handle1(bytes, dest, srcs[src_off]); \ + else if (this_cnt == 2) \ + _handle2(bytes, dest, srcs[src_off], \ + srcs[src_off + 1]); \ + else if (this_cnt == 3) \ + _handle3(bytes, dest, srcs[src_off], \ + srcs[src_off + 1], srcs[src_off + 2]); \ + else \ + _handle4(bytes, dest, srcs[src_off], \ + srcs[src_off + 1], srcs[src_off + 2], \ + srcs[src_off + 3]); \ + \ + src_cnt -= this_cnt; \ + src_off += this_cnt; \ + } \ +} + +#define DO_XOR_BLOCKS(_name, _handle1, _handle2, _handle3, _handle4) \ + static __DO_XOR_BLOCKS(_name, _handle1, _handle2, _handle3, _handle4) + +/* generic implementations */ +extern struct xor_block_template xor_block_8regs; +extern struct xor_block_template xor_block_32regs; +extern struct xor_block_template xor_block_8regs_p; +extern struct xor_block_template xor_block_32regs_p; + +void __init xor_register(struct xor_block_template *tmpl); +void __init xor_force(struct xor_block_template *tmpl); + +#endif /* _XOR_IMPL_H */ diff --git a/lib/scatterlist.c b/lib/scatterlist.c index d773720d11bf..b7fe91ef35b8 100644 --- a/lib/scatterlist.c +++ b/lib/scatterlist.c @@ -1123,8 +1123,7 @@ static ssize_t extract_user_to_sg(struct iov_iter *iter, size_t len, off; /* We decant the page list into the tail of the scatterlist */ - pages = (void *)sgtable->sgl + - array_size(sg_max, sizeof(struct scatterlist)); + pages = (void *)sg + array_size(sg_max, sizeof(struct scatterlist)); pages -= sg_max; do { @@ -1247,7 +1246,7 @@ static ssize_t extract_kvec_to_sg(struct iov_iter *iter, else page = virt_to_page((void *)kaddr); - sg_set_page(sg, page, len, off); + sg_set_page(sg, page, seg, off); sgtable->nents++; sg++; sg_max--; @@ -1256,6 +1255,7 @@ static ssize_t extract_kvec_to_sg(struct iov_iter *iter, kaddr += PAGE_SIZE; off = 0; } while (len > 0 && sg_max > 0); + ret -= len; if (maxsize <= 0 || sg_max == 0) break; @@ -1409,7 +1409,7 @@ ssize_t extract_iter_to_sg(struct iov_iter *iter, size_t maxsize, struct sg_table *sgtable, unsigned int sg_max, iov_iter_extraction_t extraction_flags) { - if (maxsize == 0) + if (maxsize == 0 || sg_max == 0) return 0; switch (iov_iter_type(iter)) { diff --git a/lib/tests/kunit_iov_iter.c b/lib/tests/kunit_iov_iter.c index bb847e5010eb..37bd6eb25896 100644 --- a/lib/tests/kunit_iov_iter.c +++ b/lib/tests/kunit_iov_iter.c @@ -13,6 +13,9 @@ #include #include #include +#include +#include +#include #include MODULE_DESCRIPTION("iov_iter testing"); @@ -37,12 +40,12 @@ static const struct kvec_test_range kvec_test_ranges[] = { static inline u8 pattern(unsigned long x) { - return x & 0xff; + return (u8)x + (u8)(x >> 8) + (u8)(x >> 16); } static void iov_kunit_unmap(void *data) { - vunmap(data); + vfree(data); } static void *__init iov_kunit_create_buffer(struct kunit *test, @@ -52,18 +55,27 @@ static void *__init iov_kunit_create_buffer(struct kunit *test, struct page **pages; unsigned long got; void *buffer; + unsigned int i; - pages = kunit_kcalloc(test, npages, sizeof(struct page *), GFP_KERNEL); - KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pages); + pages = kzalloc_objs(struct page *, npages, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pages); *ppages = pages; got = alloc_pages_bulk(GFP_KERNEL, npages, pages); if (got != npages) { release_pages(pages, got); + kvfree(pages); KUNIT_ASSERT_EQ(test, got, npages); } + /* Make sure that we don't get a physically contiguous buffer. */ + for (i = 0; i < npages / 4; ++i) + swap(pages[i], pages[i + npages / 2]); buffer = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + if (buffer == NULL) { + release_pages(pages, got); + kvfree(pages); + } KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buffer); kunit_add_action_or_reset(test, iov_kunit_unmap, buffer); @@ -369,9 +381,6 @@ static void iov_kunit_destroy_folioq(void *data) for (folioq = data; folioq; folioq = next) { next = folioq->next; - for (int i = 0; i < folioq_nr_slots(folioq); i++) - if (folioq_folio(folioq, i)) - folio_put(folioq_folio(folioq, i)); kfree(folioq); } } @@ -1009,6 +1018,202 @@ stop: KUNIT_SUCCEED(test); } +struct iov_kunit_iter_to_sg_data { + struct sg_table *sgt; + u8 *buffer, *scratch; + u8 __user *ubuf; + struct page **pages; + size_t npages; +}; + +static void __init +iov_kunit_iter_unpin_sgt(void *data) +{ + struct sg_table *sgt = data; + + for (unsigned int i = 0; i < sgt->nents; ++i) + unpin_user_page(sg_page(&sgt->sgl[i])); +} + +static void __init +iov_kunit_iter_to_sg_init(struct kunit *test, size_t bufsize, bool user, + struct iov_kunit_iter_to_sg_data *data) +{ + struct page **spages; + struct scatterlist *sg; + unsigned long uaddr; + size_t i; + + data->npages = bufsize / PAGE_SIZE; + sg = kunit_kmalloc_array(test, data->npages, sizeof(*sg), GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, sg); + sg_init_table(sg, data->npages); + data->sgt = kunit_kzalloc(test, sizeof(*data->sgt), GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, data->sgt); + data->sgt->orig_nents = 0; + data->sgt->sgl = sg; + + data->buffer = NULL; + data->ubuf = NULL; + if (user) { + uaddr = kunit_vm_mmap(test, NULL, 0, bufsize, + PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, 0); + KUNIT_ASSERT_NE(test, uaddr, 0); + data->ubuf = (u8 __user *)uaddr; + for (i = 0; i < bufsize; ++i) + put_user(pattern(i), data->ubuf + i); + } else { + data->buffer = iov_kunit_create_buffer(test, &data->pages, + data->npages); + for (i = 0; i < bufsize; ++i) + data->buffer[i] = pattern(i); + } + data->scratch = iov_kunit_create_buffer(test, &spages, data->npages); + memset(data->scratch, 0, bufsize); +} + +static void __init +iov_kunit_iter_to_sg_check(struct kunit *test, struct iov_iter *iter, + size_t bufsize, + struct iov_kunit_iter_to_sg_data *data) +{ + static const size_t tail = 16 * PAGE_SIZE; + size_t i; + + KUNIT_ASSERT_LT(test, tail, bufsize); + + if (iov_iter_extract_will_pin(iter)) + kunit_add_action_or_reset(test, iov_kunit_iter_unpin_sgt, + data->sgt); + + i = extract_iter_to_sg(iter, bufsize, data->sgt, 0, 0); + KUNIT_ASSERT_EQ(test, i, 0); + KUNIT_ASSERT_EQ(test, data->sgt->nents, 0); + + i = extract_iter_to_sg(iter, bufsize - tail, data->sgt, 1, 0); + KUNIT_ASSERT_LE(test, i, bufsize - tail); + KUNIT_ASSERT_EQ(test, data->sgt->nents, 1); + + i += extract_iter_to_sg(iter, bufsize - tail - i, data->sgt, + data->npages - data->sgt->nents, 0); + KUNIT_ASSERT_EQ(test, i, bufsize - tail); + KUNIT_ASSERT_LE(test, data->sgt->nents, data->npages); + + i += extract_iter_to_sg(iter, tail, data->sgt, + data->npages - data->sgt->nents, 0); + KUNIT_ASSERT_EQ(test, i, bufsize); + KUNIT_ASSERT_LE(test, data->sgt->nents, data->npages); + + sg_mark_end(&data->sgt->sgl[data->sgt->nents - 1]); + + i = sg_copy_to_buffer(data->sgt->sgl, data->sgt->nents, + data->scratch, bufsize); + KUNIT_ASSERT_EQ(test, i, bufsize); + + for (i = 0; i < bufsize; ++i) { + KUNIT_EXPECT_EQ_MSG(test, data->scratch[i], pattern(i), + "at i=%zx", i); + if (data->scratch[i] != pattern(i)) + break; + } + + KUNIT_EXPECT_EQ(test, i, bufsize); +} + +static void __init iov_kunit_iter_to_sg_kvec(struct kunit *test) +{ + struct iov_kunit_iter_to_sg_data data; + struct iov_iter iter; + struct kvec kvec; + size_t bufsize; + + bufsize = 0x100000; + iov_kunit_iter_to_sg_init(test, bufsize, false, &data); + + kvec.iov_base = data.buffer; + kvec.iov_len = bufsize; + iov_iter_kvec(&iter, READ, &kvec, 1, bufsize); + + iov_kunit_iter_to_sg_check(test, &iter, bufsize, &data); +} + +static void __init iov_kunit_iter_to_sg_bvec(struct kunit *test) +{ + struct iov_kunit_iter_to_sg_data data; + struct page *p, *can_merge = NULL; + size_t i, k, bufsize; + struct bio_vec *bvec; + struct iov_iter iter; + + bufsize = 0x100000; + iov_kunit_iter_to_sg_init(test, bufsize, false, &data); + + bvec = kunit_kmalloc_array(test, data.npages, sizeof(*bvec), + GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bvec); + k = 0; + for (i = 0; i < data.npages; ++i) { + p = data.pages[i]; + if (p == can_merge) + bvec[k-1].bv_len += PAGE_SIZE; + else + bvec_set_page(&bvec[k++], p, PAGE_SIZE, 0); + can_merge = p + 1; + } + iov_iter_bvec(&iter, READ, bvec, k, bufsize); + + iov_kunit_iter_to_sg_check(test, &iter, bufsize, &data); +} + +static void __init iov_kunit_iter_to_sg_folioq(struct kunit *test) +{ + struct iov_kunit_iter_to_sg_data data; + struct folio_queue *folioq; + struct iov_iter iter; + size_t bufsize; + + bufsize = 0x100000; + iov_kunit_iter_to_sg_init(test, bufsize, false, &data); + + folioq = iov_kunit_create_folioq(test); + iov_kunit_load_folioq(test, &iter, READ, folioq, data.pages, + data.npages); + + iov_kunit_iter_to_sg_check(test, &iter, bufsize, &data); +} + +static void __init iov_kunit_iter_to_sg_xarray(struct kunit *test) +{ + struct iov_kunit_iter_to_sg_data data; + struct xarray *xarray; + struct iov_iter iter; + size_t bufsize; + + bufsize = 0x100000; + iov_kunit_iter_to_sg_init(test, bufsize, false, &data); + + xarray = iov_kunit_create_xarray(test); + iov_kunit_load_xarray(test, &iter, READ, xarray, data.pages, + data.npages); + + iov_kunit_iter_to_sg_check(test, &iter, bufsize, &data); +} + +static void __init iov_kunit_iter_to_sg_ubuf(struct kunit *test) +{ + struct iov_kunit_iter_to_sg_data data; + struct iov_iter iter; + size_t bufsize; + + bufsize = 0x100000; + iov_kunit_iter_to_sg_init(test, bufsize, true, &data); + + iov_iter_ubuf(&iter, READ, data.ubuf, bufsize); + + iov_kunit_iter_to_sg_check(test, &iter, bufsize, &data); +} + static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_copy_to_kvec), KUNIT_CASE(iov_kunit_copy_from_kvec), @@ -1022,6 +1227,11 @@ static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_extract_pages_bvec), KUNIT_CASE(iov_kunit_extract_pages_folioq), KUNIT_CASE(iov_kunit_extract_pages_xarray), + KUNIT_CASE(iov_kunit_iter_to_sg_kvec), + KUNIT_CASE(iov_kunit_iter_to_sg_bvec), + KUNIT_CASE(iov_kunit_iter_to_sg_folioq), + KUNIT_CASE(iov_kunit_iter_to_sg_xarray), + KUNIT_CASE(iov_kunit_iter_to_sg_ubuf), {} }; diff --git a/lib/ts_bm.c b/lib/ts_bm.c index eed5967238c5..676105e84005 100644 --- a/lib/ts_bm.c +++ b/lib/ts_bm.c @@ -163,8 +163,22 @@ static struct ts_config *bm_init(const void *pattern, unsigned int len, struct ts_config *conf; struct ts_bm *bm; int i; - unsigned int prefix_tbl_len = len * sizeof(unsigned int); - size_t priv_size = sizeof(*bm) + len + prefix_tbl_len; + unsigned int prefix_tbl_len; + size_t priv_size; + + /* Zero-length patterns would underflow bm_find()'s initial shift. */ + if (unlikely(!len)) + return ERR_PTR(-EINVAL); + + /* + * bm->pattern is stored immediately after the good_shift[] table. + * Reject lengths that would wrap while sizing either region. + */ + if (unlikely(check_mul_overflow(len, sizeof(*bm->good_shift), + &prefix_tbl_len) || + check_add_overflow(sizeof(*bm), (size_t)len, &priv_size) || + check_add_overflow(priv_size, prefix_tbl_len, &priv_size))) + return ERR_PTR(-EINVAL); conf = alloc_ts_config(priv_size, gfp_mask); if (IS_ERR(conf)) diff --git a/lib/ts_kmp.c b/lib/ts_kmp.c index 5520dc28255a..29466c1803c9 100644 --- a/lib/ts_kmp.c +++ b/lib/ts_kmp.c @@ -94,8 +94,22 @@ static struct ts_config *kmp_init(const void *pattern, unsigned int len, struct ts_config *conf; struct ts_kmp *kmp; int i; - unsigned int prefix_tbl_len = len * sizeof(unsigned int); - size_t priv_size = sizeof(*kmp) + len + prefix_tbl_len; + unsigned int prefix_tbl_len; + size_t priv_size; + + /* Zero-length patterns would make kmp_find() read beyond kmp->pattern. */ + if (unlikely(!len)) + return ERR_PTR(-EINVAL); + + /* + * kmp->pattern is stored immediately after the prefix_tbl[] table. + * Reject lengths that would wrap while sizing either region. + */ + if (unlikely(check_mul_overflow(len, sizeof(*kmp->prefix_tbl), + &prefix_tbl_len) || + check_add_overflow(sizeof(*kmp), (size_t)len, &priv_size) || + check_add_overflow(priv_size, prefix_tbl_len, &priv_size))) + return ERR_PTR(-EINVAL); conf = alloc_ts_config(priv_size, gfp_mask); if (IS_ERR(conf)) diff --git a/lib/uuid.c b/lib/uuid.c index e8543c668dc7..128a51f1879b 100644 --- a/lib/uuid.c +++ b/lib/uuid.c @@ -54,7 +54,7 @@ EXPORT_SYMBOL(generate_random_guid); static void __uuid_gen_common(__u8 b[16]) { get_random_bytes(b, 16); - /* reversion 0b10 */ + /* revision 0b10 */ b[8] = (b[8] & 0x3F) | 0x80; } diff --git a/scripts/bloat-o-meter b/scripts/bloat-o-meter index db5dd18dc2d5..9b4fb996d95b 100755 --- a/scripts/bloat-o-meter +++ b/scripts/bloat-o-meter @@ -18,8 +18,8 @@ group.add_argument('-c', help='categorize output based on symbol type', action=' group.add_argument('-d', help='Show delta of Data Section', action='store_true') group.add_argument('-t', help='Show delta of text Section', action='store_true') parser.add_argument('-p', dest='prefix', help='Arch prefix for the tool being used. Useful in cross build scenarios') -parser.add_argument('file1', help='First file to compare') -parser.add_argument('file2', help='Second file to compare') +parser.add_argument('file_old', help='First file to compare') +parser.add_argument('file_new', help='Second file to compare') args = parser.parse_args() @@ -86,7 +86,7 @@ def calc(oldfile, newfile, format): def print_result(symboltype, symbolformat): grow, shrink, add, remove, up, down, delta, old, new, otot, ntot = \ - calc(args.file1, args.file2, symbolformat) + calc(args.file_old, args.file_new, symbolformat) print("add/remove: %s/%s grow/shrink: %s/%s up/down: %s/%s (%s)" % \ (add, remove, grow, shrink, up, -down, up-down)) diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index e56374662ff7..7e612d3e2c1a 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -641,6 +641,7 @@ our $signature_tags = qr{(?xi: Reviewed-by:| Reported-by:| Suggested-by:| + Assisted-by:| To:| Cc: )}; @@ -3105,6 +3106,15 @@ sub process { } } + # Assisted-by uses AGENT_NAME:MODEL_VERSION format, not email + if ($sign_off =~ /^Assisted-by:/i) { + if ($email !~ /^\S+:\S+/) { + WARN("BAD_SIGN_OFF", + "Assisted-by expects 'AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]' format\n" . $herecurr); + } + next; + } + my ($email_name, $name_comment, $email_address, $comment) = parse_email($email); my $suggested_email = format_email(($email_name, $name_comment, $email_address, $comment)); if ($suggested_email eq "") { @@ -7502,10 +7512,10 @@ sub process { } # check for various structs that are normally const (ops, kgdb, device_tree) -# and avoid what seem like struct definitions 'struct foo {' +# and avoid what seem like struct definitions 'struct foo {' or forward declarations 'struct foo;' if (defined($const_structs) && $line !~ /\bconst\b/ && - $line =~ /\bstruct\s+($const_structs)\b(?!\s*\{)/) { + $line =~ /\bstruct\s+($const_structs)\b(?!\s*[\{;])/) { WARN("CONST_STRUCT", "struct $1 should normally be const\n" . $herecurr); } diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh index 8d01b741de62..39d60d477bf3 100755 --- a/scripts/decode_stacktrace.sh +++ b/scripts/decode_stacktrace.sh @@ -5,9 +5,11 @@ usage() { echo "Usage:" - echo " $0 -r " - echo " $0 [ [|auto []]]" + echo " $0 [-R] -r " + echo " $0 [-R] [ [|auto []]]" echo " $0 -h" + echo "Options:" + echo " -R: decode return address instead of caller address." } # Try to find a Rust demangler @@ -33,11 +35,17 @@ fi READELF=${UTIL_PREFIX}readelf${UTIL_SUFFIX} ADDR2LINE=${UTIL_PREFIX}addr2line${UTIL_SUFFIX} NM=${UTIL_PREFIX}nm${UTIL_SUFFIX} +decode_retaddr=false if [[ $1 == "-h" ]] ; then usage exit 0 -elif [[ $1 == "-r" ]] ; then +elif [[ $1 == "-R" ]] ; then + decode_retaddr=true + shift 1 +fi + +if [[ $1 == "-r" ]] ; then vmlinux="" basepath="auto" modpath="" @@ -176,13 +184,23 @@ parse_symbol() { # Let's start doing the math to get the exact address into the # symbol. First, strip out the symbol total length. local expr=${symbol%/*} + # Also parse the offset from symbol. + local offset=${expr#*+} + offset=$((offset)) # Now, replace the symbol name with the base address we found # before. expr=${expr/$name/0x$base_addr} # Evaluate it to find the actual address - expr=$((expr)) + # The stack trace shows the return address, which is the next + # instruction after the actual call, so as long as it's in the same + # symbol, subtract one from that to point the call instruction. + if [[ $decode_retaddr == false && $offset != 0 ]]; then + expr=$((expr-1)) + else + expr=$((expr)) + fi local address=$(printf "%x\n" "$expr") # Pass it to addr2line to get filename and line number diff --git a/scripts/decodecode b/scripts/decodecode index 6364218b2178..01d25dc110de 100755 --- a/scripts/decodecode +++ b/scripts/decodecode @@ -12,7 +12,6 @@ faultlinenum=1 cleanup() { rm -f $T $T.s $T.o $T.oo $T.aa $T.dis - exit 1 } die() { @@ -49,7 +48,7 @@ done if [ -z "$code" ]; then rm $T - exit + die "Code line not found" fi echo $code diff --git a/scripts/gdb/linux/symbols.py b/scripts/gdb/linux/symbols.py index d4308b726183..943ff1228b48 100644 --- a/scripts/gdb/linux/symbols.py +++ b/scripts/gdb/linux/symbols.py @@ -298,7 +298,7 @@ are loaded as well.""" if p == "-bpf": monitor_bpf = True else: - p.append(os.path.abspath(os.path.expanduser(p))) + self.module_paths.append(os.path.abspath(os.path.expanduser(p))) self.module_paths.append(os.getcwd()) if self.breakpoint is not None: diff --git a/scripts/get_maintainer.pl b/scripts/get_maintainer.pl index 4414194bedcf..f0ca0db6ddc2 100755 --- a/scripts/get_maintainer.pl +++ b/scripts/get_maintainer.pl @@ -375,8 +375,10 @@ sub read_maintainer_file { ##Filename pattern matching if ($type eq "F" || $type eq "X") { $value =~ s@\.@\\\.@g; ##Convert . to \. + $value =~ s/\*\*/\x00/g; ##Convert ** to placeholder $value =~ s/\*/\.\*/g; ##Convert * to .* $value =~ s/\?/\./g; ##Convert ? to . + $value =~ s/\x00/(?:.*)/g; ##Convert placeholder to (?:.*) ##if pattern is a directory and it lacks a trailing slash, add one if ((-d $value)) { $value =~ s@([^/])$@$1/@; @@ -746,8 +748,10 @@ sub self_test { if (($type eq "F" || $type eq "X") && ($self_test eq "" || $self_test =~ /\bpatterns\b/)) { $value =~ s@\.@\\\.@g; ##Convert . to \. + $value =~ s/\*\*/\x00/g; ##Convert ** to placeholder $value =~ s/\*/\.\*/g; ##Convert * to .* $value =~ s/\?/\./g; ##Convert ? to . + $value =~ s/\x00/(?:.*)/g; ##Convert placeholder to (?:.*) ##if pattern is a directory and it lacks a trailing slash, add one if ((-d $value)) { $value =~ s@([^/])$@$1/@; @@ -921,7 +925,7 @@ sub get_maintainers { my $value_pd = ($value =~ tr@/@@); my $file_pd = ($file =~ tr@/@@); $value_pd++ if (substr($value,-1,1) ne "/"); - $value_pd = -1 if ($value =~ /^\.\*/); + $value_pd = -1 if ($value =~ /^(\.\*|\(\?:\.\*\))/); if ($value_pd >= $file_pd && range_is_maintained($start, $end) && range_has_maintainer($start, $end)) { @@ -955,6 +959,7 @@ sub get_maintainers { $line =~ s/([^\\])\.([^\*])/$1\?$2/g; $line =~ s/([^\\])\.$/$1\?/g; ##Convert . back to ? $line =~ s/\\\./\./g; ##Convert \. to . + $line =~ s/\(\?:\.\*\)/\*\*/g; ##Convert (?:.*) to ** $line =~ s/\.\*/\*/g; ##Convert .* to * } my $count = $line =~ s/^([A-Z]):/$1:\t/g; @@ -1048,7 +1053,7 @@ sub file_match_pattern { if ($file =~ m@^$pattern@) { my $s1 = ($file =~ tr@/@@); my $s2 = ($pattern =~ tr@/@@); - if ($s1 == $s2) { + if ($s1 == $s2 || $pattern =~ /\(\?:/) { return 1; } } diff --git a/scripts/spelling.txt b/scripts/spelling.txt index 1e89b92c2f9a..2f2e81dbda03 100644 --- a/scripts/spelling.txt +++ b/scripts/spelling.txt @@ -57,8 +57,8 @@ acknowledgement||acknowledgment ackowledge||acknowledge ackowledged||acknowledged acording||according -activete||activate actived||activated +activete||activate actualy||actually actvie||active acumulating||accumulating @@ -66,12 +66,12 @@ acumulative||accumulative acumulator||accumulator acutally||actually adapater||adapter +adddress||address adderted||asserted addional||additional additionaly||additionally additonal||additional addres||address -adddress||address addreses||addresses addresss||address addrress||address @@ -95,9 +95,9 @@ alegorical||allegorical algined||aligned algorith||algorithm algorithmical||algorithmically +algorithmn||algorithm algoritm||algorithm algoritms||algorithms -algorithmn||algorithm algorrithm||algorithm algorritm||algorithm aligment||alignment @@ -128,20 +128,20 @@ amount of times||number of times amout||amount amplifer||amplifier amplifyer||amplifier -an union||a union -an user||a user -an userspace||a userspace -an one||a one analysator||analyzer ang||and anniversery||anniversary annoucement||announcement anomolies||anomalies anomoly||anomaly +an one||a one anonynous||anonymous +an union||a union +an user||a user +an userspace||a userspace anway||anyway -aplication||application apeared||appeared +aplication||application appearence||appearance applicaion||application appliction||application @@ -155,8 +155,8 @@ approriately||appropriately apropriate||appropriate aquainted||acquainted aquired||acquired -aquisition||acquisition aquires||acquires +aquisition||acquisition arbitary||arbitrary architechture||architecture archtecture||architecture @@ -189,30 +189,30 @@ assum||assume assumtpion||assumption asume||assume asuming||assuming -asycronous||asynchronous asychronous||asynchronous -asynchnous||asynchronous -asynchrnous||asynchronous -asynchronus||asynchronous -asynchromous||asynchronous +asycronous||asynchronous asymetric||asymmetric asymmeric||asymmetric +asynchnous||asynchronous +asynchrnous||asynchronous +asynchromous||asynchronous +asynchronus||asynchronous +atempt||attempt atleast||at least atomatically||automatically atomicly||atomically -atempt||attempt atrributes||attributes attachement||attachment attatch||attach attched||attached attemp||attempt -attemps||attempts attemping||attempting +attemps||attempts attepmpt||attempt attnetion||attention attruibutes||attributes -authentification||authentication authenicated||authenticated +authentification||authentication automaticaly||automatically automaticly||automatically automatize||automate @@ -257,6 +257,7 @@ begining||beginning beter||better betweeen||between bianries||binaries +binded||bound bitmast||bitmask bitwiedh||bitwidth boardcast||broadcast @@ -287,19 +288,19 @@ calucate||calculate calulate||calculate cancelation||cancellation cancle||cancel -cant||can't -cant'||can't -canot||cannot -cann't||can't cannnot||cannot +cann't||can't +canot||cannot +cant'||can't +cant||can't capabiity||capability capabilites||capabilities capabilties||capabilities capabilty||capability capabitilies||capabilities capablity||capability -capatibilities||capabilities capapbilities||capabilities +capatibilities||capabilities captuer||capture caputure||capture carefuly||carefully @@ -307,9 +308,9 @@ cariage||carriage casued||caused catagory||category cehck||check +chache||cache challange||challenge challanges||challenges -chache||cache chanell||channel changable||changeable chanined||chained @@ -347,6 +348,7 @@ colescing||coalescing collapsable||collapsible colorfull||colorful comand||command +comaptible||compatible comit||commit commerical||commercial comming||coming @@ -357,10 +359,6 @@ committ||commit commmand||command commnunication||communication commoditiy||commodity -comsume||consume -comsumer||consumer -comsuming||consuming -comaptible||compatible compability||compatibility compaibility||compatibility comparsion||comparison @@ -376,22 +374,25 @@ compleatly||completely completition||completion completly||completely complient||compliant -componnents||components compoment||component +componnents||components comppatible||compatible compres||compress compresion||compression compresser||compressor comression||compression +comsume||consume comsumed||consumed +comsumer||consumer +comsuming||consuming comunicate||communicate comunication||communication conbination||combination concurent||concurrent conditionaly||conditionally conditon||condition -condtion||condition condtional||conditional +condtion||condition conected||connected conector||connector configed||configured @@ -428,13 +429,13 @@ continous||continuous continously||continuously continueing||continuing contiuous||continuous -contraints||constraints -contruct||construct contol||control contoller||controller +contraints||constraints controled||controlled controler||controller controll||control +contruct||construct contruction||construction contry||country conuntry||country @@ -465,10 +466,9 @@ debouce||debounce decendant||descendant decendants||descendants decompres||decompress -decsribed||described decrese||decrease decription||description -detault||default +decsribed||described dectected||detected defailt||default deferal||deferral @@ -482,9 +482,9 @@ defintion||definition defintions||definitions defualt||default defult||default -deintializing||deinitializing -deintialize||deinitialize deintialized||deinitialized +deintialize||deinitialize +deintializing||deinitializing deivce||device delared||declared delare||declare @@ -494,8 +494,8 @@ delemiter||delimiter deley||delay delibrately||deliberately delievered||delivered -demodualtor||demodulator demension||dimension +demodualtor||demodulator dependancies||dependencies dependancy||dependency dependant||dependent @@ -505,15 +505,15 @@ depreacte||deprecate desactivate||deactivate desciptor||descriptor desciptors||descriptors -descritpor||descriptor descripto||descriptor descripton||description descrition||description +descritpor||descriptor descritptor||descriptor desctiptor||descriptor +desination||destination desriptor||descriptor desriptors||descriptors -desination||destination destionation||destination destoried||destroyed destory||destroy @@ -521,6 +521,7 @@ destoryed||destroyed destorys||destroys destroied||destroyed detabase||database +detault||default deteced||detected detecion||detection detectt||detect @@ -535,55 +536,54 @@ deveolpment||development devided||divided deviece||device devision||division -diable||disable diabled||disabled +diable||disable dicline||decline +diconnected||disconnected dictionnary||dictionary didnt||didn't diferent||different -differrence||difference -diffrent||different differenciate||differentiate +differrence||difference diffreential||differential +diffrent||different diffrentiate||differentiate difinition||definition digial||digital dimention||dimension dimesions||dimensions -diconnected||disconnected -disabed||disabled -disasembler||disassembler -disble||disable -disgest||digest -disired||desired -dispalying||displaying -dissable||disable -dissapeared||disappeared diplay||display -directon||direction direcly||directly +directon||direction direectly||directly diregard||disregard -disassocation||disassociation -disassocative||disassociative +disabed||disabled disapear||disappear disapeared||disappeared disappared||disappeared -disbale||disable +disasembler||disassembler +disassocation||disassociation +disassocative||disassociative disbaled||disabled -disble||disable +disbale||disable disbled||disabled +disble||disable +disble||disable disconnet||disconnect discontinous||discontinuous +disgest||digest disharge||discharge +disired||desired disnabled||disabled +dispalying||displaying dispertion||dispersion +dissable||disable +dissapeared||disappeared dissapears||disappears dissconect||disconnect distiction||distinction divisable||divisible divsiors||divisors -dsiabled||disabled docuentation||documentation documantation||documentation documentaion||documentation @@ -598,6 +598,7 @@ downlads||downloads droped||dropped droput||dropout druing||during +dsiabled||disabled dyanmic||dynamic dynmaic||dynamic eanable||enable @@ -621,20 +622,20 @@ enble||enable enchanced||enhanced encorporating||incorporating encrupted||encrypted -encrypiton||encryption encryped||encrypted +encrypiton||encryption encryptio||encryption endianess||endianness -enpoint||endpoint enhaced||enhanced enlightnment||enlightenment +enocded||encoded +enought||enough +enpoint||endpoint enqueing||enqueuing +enterily||entirely entires||entries entites||entities entrys||entries -enocded||encoded -enought||enough -enterily||entirely enviroiment||environment enviroment||environment environement||environment @@ -653,8 +654,9 @@ evalute||evaluate evalutes||evaluates evalution||evaluation evaulated||evaluated -excecutable||executable +exaclty||exactly excceed||exceed +excecutable||executable exceded||exceeded exceds||exceeds exceeed||exceed @@ -668,41 +670,41 @@ exeuction||execution existance||existence existant||existent exixt||exist -exsits||exists exlcude||exclude exlcuding||excluding exlcusive||exclusive -exlusive||exclusive exlicitly||explicitly +exlusive||exclusive exmaple||example expecially||especially experies||expires explicite||explicit -explicity||explicitly explicitely||explicitly -explict||explicit +explicity||explicitly explictely||explicitly +explict||explicit explictly||explicitly expresion||expression exprienced||experienced exprimental||experimental -extened||extended +exsits||exists exteneded||extended +extened||extended extensability||extensibility -extention||extension extenstion||extension +extention||extension extracter||extractor faied||failed faield||failed -faild||failed failded||failed +faild||failed failer||failure -faill||fail failied||failed +faill||fail faillure||failure +failng||failing failue||failure failuer||failure -failng||failing faireness||fairness falied||failed faliure||failure @@ -717,15 +719,15 @@ fetcing||fetching fileystem||filesystem fimrware||firmware fimware||firmware +finanize||finalize +findn||find +finilizes||finalizes +finsih||finish firmare||firmware firmaware||firmware firtly||firstly firware||firmware firwmare||firmware -finanize||finalize -findn||find -finilizes||finalizes -finsih||finish fliter||filter flusing||flushing folloing||following @@ -742,9 +744,9 @@ forwared||forwarded frambuffer||framebuffer framming||framing framwork||framework +frequancy||frequency frequence||frequency frequncy||frequency -frequancy||frequency frome||from fronend||frontend fucntion||function @@ -766,9 +768,9 @@ gatable||gateable gateing||gating gauage||gauge gaurenteed||guaranteed -generiously||generously genereate||generate genereted||generated +generiously||generously genric||generic gerenal||general geting||getting @@ -790,18 +792,17 @@ hanlde||handle hanled||handled happend||happened hardare||hardware -harware||hardware hardward||hardware +harware||hardware havind||having +hearbeat||heartbeat heigth||height +heirachy||hierarchy heirarchically||hierarchically heirarchy||hierarchy -heirachy||hierarchy helpfull||helpful -hearbeat||heartbeat heterogenous||heterogeneous hexdecimal||hexadecimal -hybernate||hibernate hiearchy||hierarchy hierachy||hierarchy hierarchie||hierarchy @@ -809,14 +810,14 @@ homogenous||homogeneous horizental||horizontal howver||however hsould||should +hybernate||hibernate hypervior||hypervisor hypter||hyper idel||idle identidier||identifier iligal||illegal -illigal||illegal illgal||illegal -iomaped||iomapped +illigal||illegal imblance||imbalance immeadiately||immediately immedaite||immediate @@ -831,13 +832,14 @@ implemantation||implementation implemenation||implementation implementaiton||implementation implementated||implemented -implemention||implementation implementd||implemented +implemention||implementation implemetation||implementation implemntation||implementation implentation||implementation implmentation||implementation implmenting||implementing +inavlid||invalid incative||inactive incomming||incoming incompaitiblity||incompatibility @@ -869,9 +871,9 @@ infromation||information ingore||ignore inheritence||inheritance inital||initial -initalized||initialized initalised||initialized initalise||initialize +initalized||initialized initalize||initialize initation||initiation initators||initiators @@ -879,20 +881,20 @@ initialiazation||initialization initializationg||initialization initializiation||initialization initializtion||initialization -initialze||initialize initialzed||initialized +initialze||initialize initialzing||initializing initilization||initialization +initilized||initialized initilize||initialize initliaze||initialize -initilized||initialized inofficial||unofficial inrerface||interface insititute||institute instace||instance instal||install -instanciate||instantiate instanciated||instantiated +instanciate||instantiate instuments||instruments insufficent||insufficient intead||instead @@ -911,16 +913,16 @@ intergrated||integrated intermittant||intermittent internel||internal interoprability||interoperability -interuupt||interrupt -interupt||interrupt -interupts||interrupts -interurpt||interrupt interrface||interface interrrupt||interrupt interrup||interrupt interrups||interrupts interruptted||interrupted interupted||interrupted +interupt||interrupt +interupts||interrupts +interurpt||interrupt +interuupt||interrupt intiailized||initialized intial||initial intialisation||initialisation @@ -934,18 +936,18 @@ intrerrupt||interrupt intrrupt||interrupt intterrupt||interrupt intuative||intuitive -inavlid||invalid invaid||invalid invaild||invalid invailid||invalid -invald||invalid invalde||invalid +invald||invalid invalide||invalid invalidiate||invalidate invalud||invalid invididual||individual invokation||invocation invokations||invocations +iomaped||iomapped ireelevant||irrelevant irrelevent||irrelevant isnt||isn't @@ -991,11 +993,11 @@ losted||lost maangement||management machinary||machinery maibox||mailbox +mailformed||malformed maintainance||maintenance maintainence||maintenance maintan||maintain makeing||making -mailformed||malformed malplaced||misplaced malplace||misplace managable||manageable @@ -1005,21 +1007,22 @@ mangement||management manger||manager manoeuvering||maneuvering manufaucturing||manufacturing -mappping||mapping maping||mapping +mappping||mapping matchs||matches mathimatical||mathematical mathimatic||mathematic mathimatics||mathematics -maxmium||maximum maximium||maximum maxium||maximum +maxmium||maximum mechamism||mechanism mechanim||mechanism meetign||meeting memeory||memory memmber||member memoery||memory +memomry||memory memroy||memory ment||meant mergable||mergeable @@ -1036,19 +1039,19 @@ migrateable||migratable miliseconds||milliseconds millenium||millennium milliseonds||milliseconds -minimim||minimum -minium||minimum minimam||minimum +minimim||minimum minimun||minimum +minium||minimum miniumum||minimum minumum||minimum misalinged||misaligned miscelleneous||miscellaneous misformed||malformed -mispelled||misspelled -mispelt||misspelt mising||missing mismactch||mismatch +mispelled||misspelled +mispelt||misspelt missign||missing missmanaged||mismanaged missmatch||mismatch @@ -1061,18 +1064,17 @@ modifer||modifier modul||module modulues||modules momery||memory -memomry||memory monitring||monitoring monochorome||monochrome monochromo||monochrome monocrome||monochrome mopdule||module mroe||more -mulitplied||multiplied muliple||multiple -multipler||multiplier +mulitplied||multiplied multidimensionnal||multidimensional multipe||multiple +multipler||multiplier multple||multiple mumber||number muticast||multicast @@ -1094,6 +1096,7 @@ nerver||never nescessary||necessary nessessary||necessary none existent||non-existent +notfify||notify noticable||noticeable notication||notification notications||notifications @@ -1101,7 +1104,6 @@ notifcations||notifications notifed||notified notifer||notifier notity||notify -notfify||notify nubmer||number numebr||number numer||number @@ -1119,10 +1121,10 @@ occurence||occurrence occure||occurred occuring||occurring ocurrence||occurrence -offser||offset offet||offset offlaod||offload offloded||offloaded +offser||offset offseting||offsetting oflload||offload omited||omitted @@ -1141,25 +1143,25 @@ optionnal||optional optmizations||optimizations orientatied||orientated orientied||oriented -orignal||original originial||original +orignal||original orphanded||orphaned otherise||otherwise ouput||output oustanding||outstanding -overaall||overall -overhread||overhead -overlaping||overlapping oveflow||overflow +overaall||overall overflw||overflow -overlfow||overflow +overhread||overhead overide||override +overlaping||overlapping +overlfow||overflow overrided||overridden overriden||overridden overrrun||overrun overun||overrun -overwritting||overwriting overwriten||overwritten +overwritting||overwriting pacakge||package pachage||package packacge||package @@ -1169,11 +1171,11 @@ packtes||packets pakage||package paket||packet pallette||palette -paln||plan palne||plane +paln||plan paramameters||parameters -paramaters||parameters paramater||parameter +paramaters||parameters paramenters||parameters parametes||parameters parametised||parametrised @@ -1241,8 +1243,6 @@ prefered||preferred prefferably||preferably prefitler||prefilter preform||perform -previleged||privileged -previlege||privilege premption||preemption prepaired||prepared prepate||prepare @@ -1250,6 +1250,8 @@ preperation||preparation preprare||prepare pressre||pressure presuambly||presumably +previleged||privileged +previlege||privilege previosuly||previously previsously||previously primative||primitive @@ -1258,17 +1260,17 @@ priorty||priority priting||printing privilaged||privileged privilage||privilege -priviledge||privilege priviledged||privileged +priviledge||privilege priviledges||privileges privleges||privileges -probaly||probably probabalistic||probabilistic +probaly||probably procceed||proceed proccesors||processors procesed||processed -proces||process procesing||processing +proces||process processessing||processing processess||processes processpr||processor @@ -1288,6 +1290,7 @@ progresss||progress prohibitted||prohibited prohibitting||prohibiting promiscous||promiscuous +promixity||proximity promps||prompts pronnounced||pronounced prononciation||pronunciation @@ -1296,15 +1299,14 @@ pronunce||pronounce propery||property propigate||propagate propigation||propagation -propogation||propagation propogate||propagate +propogation||propagation prosess||process protable||portable protcol||protocol protecion||protection protedcted||protected protocoll||protocol -promixity||proximity psudo||pseudo psuedo||pseudo psychadelic||psychedelic @@ -1333,8 +1335,8 @@ recieves||receives recieving||receiving recogniced||recognised recognizeable||recognizable -recompte||recompute recommanded||recommended +recompte||recompute recyle||recycle redect||reject redircet||redirect @@ -1344,8 +1346,8 @@ reename||rename refcounf||refcount refence||reference refered||referred -referencce||reference referenace||reference +referencce||reference refererence||reference refering||referring refernces||references @@ -1353,8 +1355,8 @@ refernnce||reference refrence||reference regiser||register registed||registered -registerd||registered registeration||registration +registerd||registered registeresd||registered registerred||registered registes||registers @@ -1371,8 +1373,8 @@ reloade||reload remoote||remote remore||remote removeable||removable -repective||respective repectively||respectively +repective||respective replacable||replaceable replacments||replacements replys||replies @@ -1389,8 +1391,8 @@ requieres||requires requirment||requirement requred||required requried||required -requst||request requsted||requested +requst||request reregisteration||reregistration reseting||resetting reseved||reserved @@ -1412,11 +1414,11 @@ retransmited||retransmitted retreived||retrieved retreive||retrieve retreiving||retrieving -retrive||retrieve retrived||retrieved +retrive||retrieve retrun||return -retun||return retuned||returned +retun||return reudce||reduce reuest||request reuqest||request @@ -1464,9 +1466,9 @@ seperate||separate seperatly||separately seperator||separator sepperate||separate -seqeunce||sequence -seqeuncer||sequencer seqeuencer||sequencer +seqeuncer||sequencer +seqeunce||sequence sequece||sequence sequemce||sequence sequencial||sequential @@ -1505,8 +1507,8 @@ soley||solely soluation||solution souce||source speach||speech -specfic||specific specfication||specification +specfic||specific specfield||specified speciefied||specified specifc||specific @@ -1515,8 +1517,8 @@ specificatin||specification specificaton||specification specificed||specified specifing||specifying -specifiy||specify specifiying||specifying +specifiy||specify speficied||specified speicify||specify speling||spelling @@ -1543,23 +1545,23 @@ stoppped||stopped straming||streaming struc||struct structres||structures -stuct||struct strucuture||structure +stuct||struct stucture||structure sturcture||structure subdirectoires||subdirectories suble||subtle -substract||subtract submited||submitted submition||submission +substract||subtract succeded||succeeded -suceed||succeed -succesfuly||successfully succesfully||successfully succesful||successful +succesfuly||successfully successed||succeeded successfull||successful successfuly||successfully +suceed||succeed sucessfully||successfully sucessful||successful sucess||success @@ -1568,9 +1570,9 @@ superseeded||superseded suplied||supplied suported||supported suport||support -supportet||supported suppored||supported supporing||supporting +supportet||supported supportin||supporting suppoted||supported suppported||supported @@ -1581,27 +1583,27 @@ surpressed||suppressed surpresses||suppresses susbsystem||subsystem suspeneded||suspended -suspsend||suspend suspicously||suspiciously +suspsend||suspend swaping||swapping switchs||switches -swith||switch swithable||switchable -swithc||switch swithced||switched swithcing||switching +swithc||switch swithed||switched swithing||switching +swith||switch swtich||switch +sychronization||synchronization +sychronously||synchronously syfs||sysfs symetric||symmetric synax||syntax synchonized||synchronized -sychronization||synchronization -sychronously||synchronously synchronuously||synchronously -syncronize||synchronize syncronized||synchronized +syncronize||synchronize syncronizing||synchronizing syncronus||synchronous syste||system @@ -1610,16 +1612,17 @@ sythesis||synthesis tagert||target taht||that tained||tainted -tarffic||traffic +tansition||transition tansmit||transmit +tarffic||traffic targetted||targeted targetting||targeting taskelt||tasklet teh||the temeprature||temperature temorary||temporary -temproarily||temporarily temperture||temperature +temproarily||temporarily theads||threads therfore||therefore thier||their @@ -1629,23 +1632,20 @@ threshhold||threshold thresold||threshold throtting||throttling throught||through -tansition||transition -trackling||tracking -troughput||throughput -trys||tries thses||these -tiggers||triggers tiggered||triggered tiggerring||triggering -tipically||typically +tiggers||triggers timeing||timing timming||timing timout||timeout +tipically||typically tmis||this tolarance||tolerance toogle||toggle torerable||tolerable torlence||tolerance +trackling||tracking traget||target traking||tracking tramsmitted||transmitted @@ -1669,20 +1669,20 @@ trasfer||transfer trasmission||transmission trasmitter||transmitter treshold||threshold -trigged||triggered -triggerd||triggered trigerred||triggered trigerring||triggering +trigged||triggered +triggerd||triggered +troughput||throughput trun||turn +trys||tries tunning||tuning ture||true tyep||type udpate||update -updtes||updates uesd||used -unknwon||unknown uknown||unknown -usccess||success +unamed||unnamed uncommited||uncommitted uncompatible||incompatible uncomressed||uncompressed @@ -1691,6 +1691,7 @@ undeflow||underflow undelying||underlying underun||underrun unecessary||unnecessary +uneeded||unneeded unexecpted||unexpected unexepected||unexpected unexpcted||unexpected @@ -1699,26 +1700,24 @@ unexpeted||unexpected unexpexted||unexpected unfortunatelly||unfortunately unifiy||unify -uniterrupted||uninterrupted uninterruptable||uninterruptible unintialized||uninitialized +uniterrupted||uninterrupted unitialized||uninitialized unkmown||unknown unknonw||unknown unknouwn||unknown unknow||unknown +unknwon||unknown unkown||unknown -unamed||unnamed -uneeded||unneeded -unneded||unneeded +unmached||unmatched unneccecary||unnecessary unneccesary||unnecessary unneccessary||unnecessary unnecesary||unnecessary +unneded||unneeded unneedingly||unnecessarily unnsupported||unsupported -unuspported||unsupported -unmached||unmatched unprecise||imprecise unpriviledged||unprivileged unpriviliged||unprivileged @@ -1726,18 +1725,21 @@ unregester||unregister unresgister||unregister unrgesiter||unregister unsinged||unsigned -unstabel||unstable -unsolicted||unsolicited unsolicitied||unsolicited +unsolicted||unsolicited +unstabel||unstable unsuccessfull||unsuccessful unsuported||unsupported untill||until ununsed||unused unuseful||useless +unuspported||unsupported unvalid||invalid upate||update +updtes||updates upsupported||unsupported upto||up to +usccess||success useable||usable usefule||useful usefull||useful @@ -1759,14 +1761,14 @@ variantions||variations varible||variable varient||variant vaule||value -verbse||verbose veify||verify +verbse||verbose verfication||verification veriosn||version -versoin||version verisons||versions verison||version veritical||vertical +versoin||version verson||version vicefersa||vice-versa virtal||virtual @@ -1779,13 +1781,13 @@ wakeus||wakeups was't||wasn't wathdog||watchdog wating||waiting -wiat||wait wether||whether whataver||whatever whcih||which whenver||whenever wheter||whether whe||when +wiat||wait wierd||weird wihout||without wiil||will diff --git a/tools/accounting/Makefile b/tools/accounting/Makefile index 20bbd461515e..007c0bb8cbbb 100644 --- a/tools/accounting/Makefile +++ b/tools/accounting/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 CC := $(CROSS_COMPILE)gcc -CFLAGS := -I../../usr/include +CFLAGS := -I../include/uapi/ PROGS := getdelays procacct delaytop diff --git a/tools/accounting/getdelays.c b/tools/accounting/getdelays.c index 50792df27707..368a622ca027 100644 --- a/tools/accounting/getdelays.c +++ b/tools/accounting/getdelays.c @@ -60,7 +60,7 @@ int print_task_context_switch_counts; } /* Maximum size of response requested or message sent */ -#define MAX_MSG_SIZE 1024 +#define MAX_MSG_SIZE 2048 /* Maximum number of cpus expected to be specified in a cpumask */ #define MAX_CPUS 32 @@ -115,6 +115,32 @@ error: return -1; } +static int recv_taskstats_msg(int sd, struct msgtemplate *msg) +{ + struct sockaddr_nl nladdr; + struct iovec iov = { + .iov_base = msg, + .iov_len = sizeof(*msg), + }; + struct msghdr hdr = { + .msg_name = &nladdr, + .msg_namelen = sizeof(nladdr), + .msg_iov = &iov, + .msg_iovlen = 1, + }; + int ret; + + ret = recvmsg(sd, &hdr, 0); + if (ret < 0) + return -1; + if (hdr.msg_flags & MSG_TRUNC) { + errno = EMSGSIZE; + return -1; + } + + return ret; +} + static int send_cmd(int sd, __u16 nlmsg_type, __u32 nlmsg_pid, __u8 genl_cmd, __u16 nla_type, @@ -633,12 +659,16 @@ int main(int argc, char *argv[]) } do { - rep_len = recv(nl_sd, &msg, sizeof(msg), 0); + rep_len = recv_taskstats_msg(nl_sd, &msg); PRINTF("received %d bytes\n", rep_len); if (rep_len < 0) { - fprintf(stderr, "nonfatal reply error: errno %d\n", - errno); + if (errno == EMSGSIZE) + fprintf(stderr, + "dropped truncated taskstats netlink message, please increase MAX_MSG_SIZE\n"); + else + fprintf(stderr, "nonfatal reply error: errno %d\n", + errno); continue; } if (msg.n.nlmsg_type == NLMSG_ERROR || @@ -680,6 +710,9 @@ int main(int argc, char *argv[]) printf("TGID\t%d\n", rtid); break; case TASKSTATS_TYPE_STATS: + PRINTF("version %u\n", + ((struct taskstats *) + NLA_DATA(na))->version); if (print_delays) print_delayacct((struct taskstats *) NLA_DATA(na)); if (print_io_accounting) diff --git a/tools/accounting/procacct.c b/tools/accounting/procacct.c index e8dee05a6264..46e5986ad927 100644 --- a/tools/accounting/procacct.c +++ b/tools/accounting/procacct.c @@ -71,7 +71,7 @@ int print_task_context_switch_counts; } /* Maximum size of response requested or message sent */ -#define MAX_MSG_SIZE 1024 +#define MAX_MSG_SIZE 2048 /* Maximum number of cpus expected to be specified in a cpumask */ #define MAX_CPUS 32 @@ -121,6 +121,32 @@ error: return -1; } +static int recv_taskstats_msg(int sd, struct msgtemplate *msg) +{ + struct sockaddr_nl nladdr; + struct iovec iov = { + .iov_base = msg, + .iov_len = sizeof(*msg), + }; + struct msghdr hdr = { + .msg_name = &nladdr, + .msg_namelen = sizeof(nladdr), + .msg_iov = &iov, + .msg_iovlen = 1, + }; + int ret; + + ret = recvmsg(sd, &hdr, 0); + if (ret < 0) + return -1; + if (hdr.msg_flags & MSG_TRUNC) { + errno = EMSGSIZE; + return -1; + } + + return ret; +} + static int send_cmd(int sd, __u16 nlmsg_type, __u32 nlmsg_pid, __u8 genl_cmd, __u16 nla_type, @@ -239,6 +265,8 @@ void handle_aggr(int mother, struct nlattr *na, int fd) PRINTF("TGID\t%d\n", rtid); break; case TASKSTATS_TYPE_STATS: + PRINTF("version %u\n", + ((struct taskstats *)NLA_DATA(na))->version); if (mother == TASKSTATS_TYPE_AGGR_PID) print_procacct((struct taskstats *) NLA_DATA(na)); if (fd) { @@ -347,12 +375,16 @@ int main(int argc, char *argv[]) } do { - rep_len = recv(nl_sd, &msg, sizeof(msg), 0); + rep_len = recv_taskstats_msg(nl_sd, &msg); PRINTF("received %d bytes\n", rep_len); if (rep_len < 0) { - fprintf(stderr, "nonfatal reply error: errno %d\n", - errno); + if (errno == EMSGSIZE) + fprintf(stderr, + "dropped truncated taskstats netlink message, please increase MAX_MSG_SIZE\n"); + else + fprintf(stderr, "nonfatal reply error: errno %d\n", + errno); continue; } if (msg.n.nlmsg_type == NLMSG_ERROR || diff --git a/tools/include/uapi/linux/taskstats.h b/tools/include/uapi/linux/taskstats.h new file mode 100644 index 000000000000..3ae25f3ce067 --- /dev/null +++ b/tools/include/uapi/linux/taskstats.h @@ -0,0 +1,291 @@ +/* SPDX-License-Identifier: LGPL-2.1 WITH Linux-syscall-note */ +/* taskstats.h - exporting per-task statistics + * + * Copyright (C) Shailabh Nagar, IBM Corp. 2006 + * (C) Balbir Singh, IBM Corp. 2006 + * (C) Jay Lan, SGI, 2006 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of version 2.1 of the GNU Lesser General Public License + * as published by the Free Software Foundation. + * + * This program is distributed in the hope that it would be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + */ + +#ifndef _LINUX_TASKSTATS_H +#define _LINUX_TASKSTATS_H + +#include +#include + +/* Format for per-task data returned to userland when + * - a task exits + * - listener requests stats for a task + * + * The struct is versioned. Newer versions should only add fields to + * the bottom of the struct to maintain backward compatibility. + * + * + * To add new fields + * a) bump up TASKSTATS_VERSION + * b) add comment indicating new version number at end of struct + * c) add new fields after version comment; maintain 64-bit alignment + */ + + +#define TASKSTATS_VERSION 17 +#define TS_COMM_LEN 32 /* should be >= TASK_COMM_LEN + * in linux/sched.h */ + +struct taskstats { + + /* The version number of this struct. This field is always set to + * TAKSTATS_VERSION, which is defined in . + * Each time the struct is changed, the value should be incremented. + */ + __u16 version; + __u32 ac_exitcode; /* Exit status */ + + /* The accounting flags of a task as defined in + * Defined values are AFORK, ASU, ACOMPAT, ACORE, AXSIG, and AGROUP. + * (AGROUP since version 12). + */ + __u8 ac_flag; /* Record flags */ + __u8 ac_nice; /* task_nice */ + + /* Delay accounting fields start + * + * All values, until comment "Delay accounting fields end" are + * available only if delay accounting is enabled, even though the last + * few fields are not delays + * + * xxx_count is the number of delay values recorded + * xxx_delay_total is the corresponding cumulative delay in nanoseconds + * + * xxx_delay_total wraps around to zero on overflow + * xxx_count incremented regardless of overflow + */ + + /* Delay waiting for cpu, while runnable + * count, delay_total NOT updated atomically + */ + __u64 cpu_count __attribute__((aligned(8))); + __u64 cpu_delay_total; + + /* Following four fields atomically updated using task->delays->lock */ + + /* Delay waiting for synchronous block I/O to complete + * does not account for delays in I/O submission + */ + __u64 blkio_count; + __u64 blkio_delay_total; + + /* Delay waiting for page fault I/O (swap in only) */ + __u64 swapin_count; + __u64 swapin_delay_total; + + /* cpu "wall-clock" running time + * On some architectures, value will adjust for cpu time stolen + * from the kernel in involuntary waits due to virtualization. + * Value is cumulative, in nanoseconds, without a corresponding count + * and wraps around to zero silently on overflow + */ + __u64 cpu_run_real_total; + + /* cpu "virtual" running time + * Uses time intervals seen by the kernel i.e. no adjustment + * for kernel's involuntary waits due to virtualization. + * Value is cumulative, in nanoseconds, without a corresponding count + * and wraps around to zero silently on overflow + */ + __u64 cpu_run_virtual_total; + /* Delay accounting fields end */ + /* version 1 ends here */ + + /* Basic Accounting Fields start */ + char ac_comm[TS_COMM_LEN]; /* Command name */ + __u8 ac_sched __attribute__((aligned(8))); + /* Scheduling discipline */ + __u8 ac_pad[3]; + __u32 ac_uid __attribute__((aligned(8))); + /* User ID */ + __u32 ac_gid; /* Group ID */ + __u32 ac_pid; /* Process ID */ + __u32 ac_ppid; /* Parent process ID */ + /* __u32 range means times from 1970 to 2106 */ + __u32 ac_btime; /* Begin time [sec since 1970] */ + __u64 ac_etime __attribute__((aligned(8))); + /* Elapsed time [usec] */ + __u64 ac_utime; /* User CPU time [usec] */ + __u64 ac_stime; /* SYstem CPU time [usec] */ + __u64 ac_minflt; /* Minor Page Fault Count */ + __u64 ac_majflt; /* Major Page Fault Count */ + /* Basic Accounting Fields end */ + + /* Extended accounting fields start */ + /* Accumulated RSS usage in duration of a task, in MBytes-usecs. + * The current rss usage is added to this counter every time + * a tick is charged to a task's system time. So, at the end we + * will have memory usage multiplied by system time. Thus an + * average usage per system time unit can be calculated. + */ + __u64 coremem; /* accumulated RSS usage in MB-usec */ + /* Accumulated virtual memory usage in duration of a task. + * Same as acct_rss_mem1 above except that we keep track of VM usage. + */ + __u64 virtmem; /* accumulated VM usage in MB-usec */ + + /* High watermark of RSS and virtual memory usage in duration of + * a task, in KBytes. + */ + __u64 hiwater_rss; /* High-watermark of RSS usage, in KB */ + __u64 hiwater_vm; /* High-water VM usage, in KB */ + + /* The following four fields are I/O statistics of a task. */ + __u64 read_char; /* bytes read */ + __u64 write_char; /* bytes written */ + __u64 read_syscalls; /* read syscalls */ + __u64 write_syscalls; /* write syscalls */ + /* Extended accounting fields end */ + +#define TASKSTATS_HAS_IO_ACCOUNTING + /* Per-task storage I/O accounting starts */ + __u64 read_bytes; /* bytes of read I/O */ + __u64 write_bytes; /* bytes of write I/O */ + __u64 cancelled_write_bytes; /* bytes of cancelled write I/O */ + + __u64 nvcsw; /* voluntary_ctxt_switches */ + __u64 nivcsw; /* nonvoluntary_ctxt_switches */ + + /* time accounting for SMT machines */ + __u64 ac_utimescaled; /* utime scaled on frequency etc */ + __u64 ac_stimescaled; /* stime scaled on frequency etc */ + __u64 cpu_scaled_run_real_total; /* scaled cpu_run_real_total */ + + /* Delay waiting for memory reclaim */ + __u64 freepages_count; + __u64 freepages_delay_total; + + + /* Delay waiting for thrashing page */ + __u64 thrashing_count; + __u64 thrashing_delay_total; + + /* v10: 64-bit btime to avoid overflow */ + __u64 ac_btime64; /* 64-bit begin time */ + + /* v11: Delay waiting for memory compact */ + __u64 compact_count; + __u64 compact_delay_total; + + /* v12 begin */ + __u32 ac_tgid; /* thread group ID */ + /* Thread group walltime up to now. This is total process walltime if + * AGROUP flag is set. + */ + __u64 ac_tgetime __attribute__((aligned(8))); + /* Lightweight information to identify process binary files. + * This leaves userspace to match this to a file system path, using + * MAJOR() and MINOR() macros to identify a device and mount point, + * the inode to identify the executable file. This is /proc/self/exe + * at the end, so matching the most recent exec(). Values are zero + * for kernel threads. + */ + __u64 ac_exe_dev; /* program binary device ID */ + __u64 ac_exe_inode; /* program binary inode number */ + /* v12 end */ + + /* v13: Delay waiting for write-protect copy */ + __u64 wpcopy_count; + __u64 wpcopy_delay_total; + + /* v14: Delay waiting for IRQ/SOFTIRQ */ + __u64 irq_count; + __u64 irq_delay_total; + + /* v15: add Delay max and Delay min */ + + /* v16: move Delay max and Delay min to the end of taskstat */ + __u64 cpu_delay_max; + __u64 cpu_delay_min; + + __u64 blkio_delay_max; + __u64 blkio_delay_min; + + __u64 swapin_delay_max; + __u64 swapin_delay_min; + + __u64 freepages_delay_max; + __u64 freepages_delay_min; + + __u64 thrashing_delay_max; + __u64 thrashing_delay_min; + + __u64 compact_delay_max; + __u64 compact_delay_min; + + __u64 wpcopy_delay_max; + __u64 wpcopy_delay_min; + + __u64 irq_delay_max; + __u64 irq_delay_min; + + /*v17: delay max timestamp record*/ + struct __kernel_timespec cpu_delay_max_ts; + struct __kernel_timespec blkio_delay_max_ts; + struct __kernel_timespec swapin_delay_max_ts; + struct __kernel_timespec freepages_delay_max_ts; + struct __kernel_timespec thrashing_delay_max_ts; + struct __kernel_timespec compact_delay_max_ts; + struct __kernel_timespec wpcopy_delay_max_ts; + struct __kernel_timespec irq_delay_max_ts; +}; + + +/* + * Commands sent from userspace + * Not versioned. New commands should only be inserted at the enum's end + * prior to __TASKSTATS_CMD_MAX + */ + +enum { + TASKSTATS_CMD_UNSPEC = 0, /* Reserved */ + TASKSTATS_CMD_GET, /* user->kernel request/get-response */ + TASKSTATS_CMD_NEW, /* kernel->user event */ + __TASKSTATS_CMD_MAX, +}; + +#define TASKSTATS_CMD_MAX (__TASKSTATS_CMD_MAX - 1) + +enum { + TASKSTATS_TYPE_UNSPEC = 0, /* Reserved */ + TASKSTATS_TYPE_PID, /* Process id */ + TASKSTATS_TYPE_TGID, /* Thread group id */ + TASKSTATS_TYPE_STATS, /* taskstats structure */ + TASKSTATS_TYPE_AGGR_PID, /* contains pid + stats */ + TASKSTATS_TYPE_AGGR_TGID, /* contains tgid + stats */ + TASKSTATS_TYPE_NULL, /* contains nothing */ + __TASKSTATS_TYPE_MAX, +}; + +#define TASKSTATS_TYPE_MAX (__TASKSTATS_TYPE_MAX - 1) + +enum { + TASKSTATS_CMD_ATTR_UNSPEC = 0, + TASKSTATS_CMD_ATTR_PID, + TASKSTATS_CMD_ATTR_TGID, + TASKSTATS_CMD_ATTR_REGISTER_CPUMASK, + TASKSTATS_CMD_ATTR_DEREGISTER_CPUMASK, + __TASKSTATS_CMD_ATTR_MAX, +}; + +#define TASKSTATS_CMD_ATTR_MAX (__TASKSTATS_CMD_ATTR_MAX - 1) + +/* NETLINK_GENERIC related info */ + +#define TASKSTATS_GENL_NAME "TASKSTATS" +#define TASKSTATS_GENL_VERSION 0x1 + +#endif /* _LINUX_TASKSTATS_H */ diff --git a/tools/testing/selftests/breakpoints/Makefile b/tools/testing/selftests/breakpoints/Makefile index 9ec2c78de8ca..0b8f5acf7c78 100644 --- a/tools/testing/selftests/breakpoints/Makefile +++ b/tools/testing/selftests/breakpoints/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 # Taken from perf makefile -uname_M := $(shell uname -m 2>/dev/null || echo not) -ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/x86/ -e s/x86_64/x86/) +ARCH ?= $(shell uname -m 2>/dev/null || echo not) +override ARCH := $(shell echo $(ARCH) | sed -e s/i.86/x86/ -e s/x86_64/x86/) TEST_GEN_PROGS := step_after_suspend_test diff --git a/tools/testing/selftests/fchmodat2/fchmodat2_test.c b/tools/testing/selftests/fchmodat2/fchmodat2_test.c index e397339495f6..8e1b6384b319 100644 --- a/tools/testing/selftests/fchmodat2/fchmodat2_test.c +++ b/tools/testing/selftests/fchmodat2/fchmodat2_test.c @@ -9,6 +9,11 @@ #include "kselftest.h" +struct testdir { + char *dirname; + int dfd; +}; + int sys_fchmodat2(int dfd, const char *filename, mode_t mode, int flags) { int ret = syscall(__NR_fchmodat2, dfd, filename, mode, flags); @@ -16,9 +21,9 @@ int sys_fchmodat2(int dfd, const char *filename, mode_t mode, int flags) return ret >= 0 ? ret : -errno; } -int setup_testdir(void) +static void setup_testdir(struct testdir *testdir) { - int dfd, ret; + int ret, dfd; char dirname[] = "/tmp/ksft-fchmodat2.XXXXXX"; /* Make the top-level directory. */ @@ -26,21 +31,48 @@ int setup_testdir(void) ksft_exit_fail_msg("%s: failed to create tmpdir\n", __func__); dfd = open(dirname, O_PATH | O_DIRECTORY); - if (dfd < 0) - ksft_exit_fail_msg("%s: failed to open tmpdir\n", __func__); + if (dfd < 0) { + ksft_perror("failed to open tmpdir"); + goto err; + } ret = openat(dfd, "regfile", O_CREAT | O_WRONLY | O_TRUNC, 0644); - if (ret < 0) - ksft_exit_fail_msg("%s: failed to create file in tmpdir\n", - __func__); + if (ret < 0) { + ksft_perror("failed to create file in tmpdir"); + goto err; + } close(ret); ret = symlinkat("regfile", dfd, "symlink"); - if (ret < 0) - ksft_exit_fail_msg("%s: failed to create symlink in tmpdir\n", - __func__); + if (ret < 0) { + ksft_perror("symlinkat() failed"); + goto err_regfile; + } - return dfd; + testdir->dirname = strdup(dirname); + if (!testdir->dirname) { + ksft_perror("Out of memory"); + goto err_symlink; + } + testdir->dfd = dfd; + + return; + +err_symlink: + unlinkat(testdir->dfd, "symlink", 0); +err_regfile: + unlinkat(testdir->dfd, "regfile", 0); +err: + unlink(dirname); + ksft_exit_fail(); +} + +static void cleanup_testdir(struct testdir *testdir) +{ + unlinkat(testdir->dfd, "regfile", 0); + unlinkat(testdir->dfd, "symlink", 0); + rmdir(testdir->dirname); + free(testdir->dirname); } int expect_mode(int dfd, const char *filename, mode_t expect_mode) @@ -48,61 +80,80 @@ int expect_mode(int dfd, const char *filename, mode_t expect_mode) struct stat st; int ret = fstatat(dfd, filename, &st, AT_SYMLINK_NOFOLLOW); - if (ret) - ksft_exit_fail_msg("%s: %s: fstatat failed\n", - __func__, filename); + if (ret) { + ksft_perror("fstatat() failed\n"); + return 0; + } return (st.st_mode == expect_mode); } void test_regfile(void) { - int dfd, ret; + struct testdir testdir; + int ret; - dfd = setup_testdir(); + setup_testdir(&testdir); - ret = sys_fchmodat2(dfd, "regfile", 0640, 0); + ret = sys_fchmodat2(testdir.dfd, "regfile", 0640, 0); - if (ret < 0) - ksft_exit_fail_msg("%s: fchmodat2(noflag) failed\n", __func__); + if (ret < 0) { + ksft_perror("fchmodat2(noflag) failed"); + goto out; + } - if (!expect_mode(dfd, "regfile", 0100640)) - ksft_exit_fail_msg("%s: wrong file mode bits after fchmodat2\n", + if (!expect_mode(testdir.dfd, "regfile", 0100640)) { + ksft_print_msg("%s: wrong file mode bits after fchmodat2\n", __func__); + ret = 1; + goto out; + } - ret = sys_fchmodat2(dfd, "regfile", 0600, AT_SYMLINK_NOFOLLOW); + ret = sys_fchmodat2(testdir.dfd, "regfile", 0600, AT_SYMLINK_NOFOLLOW); - if (ret < 0) - ksft_exit_fail_msg("%s: fchmodat2(AT_SYMLINK_NOFOLLOW) failed\n", - __func__); + if (ret < 0) { + ksft_perror("fchmodat2(AT_SYMLINK_NOFOLLOW) failed"); + goto out; + } - if (!expect_mode(dfd, "regfile", 0100600)) - ksft_exit_fail_msg("%s: wrong file mode bits after fchmodat2 with nofollow\n", - __func__); + if (!expect_mode(testdir.dfd, "regfile", 0100600)) { + ksft_print_msg("%s: wrong file mode bits after fchmodat2 with nofollow\n", + __func__); + ret = 1; + } - ksft_test_result_pass("fchmodat2(regfile)\n"); +out: + ksft_test_result(ret == 0, "fchmodat2(regfile)\n"); + cleanup_testdir(&testdir); } void test_symlink(void) { - int dfd, ret; + struct testdir testdir; + int ret; - dfd = setup_testdir(); + setup_testdir(&testdir); - ret = sys_fchmodat2(dfd, "symlink", 0640, 0); + ret = sys_fchmodat2(testdir.dfd, "symlink", 0640, 0); - if (ret < 0) - ksft_exit_fail_msg("%s: fchmodat2(noflag) failed\n", __func__); + if (ret < 0) { + ksft_perror("fchmodat2(noflag) failed"); + goto err; + } - if (!expect_mode(dfd, "regfile", 0100640)) - ksft_exit_fail_msg("%s: wrong file mode bits after fchmodat2\n", - __func__); + if (!expect_mode(testdir.dfd, "regfile", 0100640)) { + ksft_print_msg("%s: wrong file mode bits after fchmodat2\n", + __func__); + goto err; + } - if (!expect_mode(dfd, "symlink", 0120777)) - ksft_exit_fail_msg("%s: wrong symlink mode bits after fchmodat2\n", - __func__); + if (!expect_mode(testdir.dfd, "symlink", 0120777)) { + ksft_print_msg("%s: wrong symlink mode bits after fchmodat2\n", + __func__); + goto err; + } - ret = sys_fchmodat2(dfd, "symlink", 0600, AT_SYMLINK_NOFOLLOW); + ret = sys_fchmodat2(testdir.dfd, "symlink", 0600, AT_SYMLINK_NOFOLLOW); /* * On certain filesystems (xfs or btrfs), chmod operation fails. So we @@ -111,18 +162,28 @@ void test_symlink(void) * * https://sourceware.org/legacy-ml/libc-alpha/2020-02/msg00467.html */ - if (ret == 0 && !expect_mode(dfd, "symlink", 0120600)) - ksft_exit_fail_msg("%s: wrong symlink mode bits after fchmodat2 with nofollow\n", + if (ret == 0 && !expect_mode(testdir.dfd, "symlink", 0120600)) { + ksft_print_msg("%s: wrong symlink mode bits after fchmodat2 with nofollow\n", __func__); + ret = 1; + goto err; + } - if (!expect_mode(dfd, "regfile", 0100640)) - ksft_exit_fail_msg("%s: wrong file mode bits after fchmodat2 with nofollow\n", - __func__); + if (!expect_mode(testdir.dfd, "regfile", 0100640)) { + ksft_print_msg("%s: wrong file mode bits after fchmodat2 with nofollow\n", + __func__); + } if (ret != 0) ksft_test_result_skip("fchmodat2(symlink)\n"); else ksft_test_result_pass("fchmodat2(symlink)\n"); + cleanup_testdir(&testdir); + return; + +err: + ksft_test_result_fail("fchmodat2(symlink)\n"); + cleanup_testdir(&testdir); } #define NUM_TESTS 2 @@ -135,8 +196,5 @@ int main(int argc, char **argv) test_regfile(); test_symlink(); - if (ksft_get_fail_cnt() + ksft_get_error_cnt() > 0) - ksft_exit_fail(); - else - ksft_exit_pass(); + ksft_finished(); } diff --git a/tools/testing/selftests/ipc/Makefile b/tools/testing/selftests/ipc/Makefile index 50e9c299fc4a..fad10f2bb57b 100644 --- a/tools/testing/selftests/ipc/Makefile +++ b/tools/testing/selftests/ipc/Makefile @@ -1,12 +1,12 @@ # SPDX-License-Identifier: GPL-2.0 -uname_M := $(shell uname -m 2>/dev/null || echo not) -ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/i386/) +ARCH ?= $(shell uname -m 2>/dev/null || echo not) +override ARCH := $(shell echo $(ARCH) | sed -e s/i.86/i386/) ifeq ($(ARCH),i386) - ARCH := x86 + override ARCH := x86 CFLAGS := -DCONFIG_X86_32 -D__i386__ endif ifeq ($(ARCH),x86_64) - ARCH := x86 + override ARCH := x86 CFLAGS := -DCONFIG_X86_64 -D__x86_64__ endif diff --git a/tools/testing/selftests/ipc/msgque.c b/tools/testing/selftests/ipc/msgque.c index e107379d185c..82f73cdae120 100644 --- a/tools/testing/selftests/ipc/msgque.c +++ b/tools/testing/selftests/ipc/msgque.c @@ -161,6 +161,9 @@ int dump_queue(struct msgque_data *msgque) ret = msgrcv(msgque->msq_id, &msgque->messages[i].mtype, MAX_MSG_SIZE, i, IPC_NOWAIT | MSG_COPY); if (ret < 0) { + if (errno == ENOSYS) + ksft_exit_skip("MSG_COPY not supported\n"); + ksft_test_result_fail("Failed to copy IPC message: %m (%d)\n", errno); return -errno; } diff --git a/tools/testing/selftests/prctl/Makefile b/tools/testing/selftests/prctl/Makefile index 01dc90fbb509..e770e86fad9a 100644 --- a/tools/testing/selftests/prctl/Makefile +++ b/tools/testing/selftests/prctl/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 ifndef CROSS_COMPILE -uname_M := $(shell uname -m 2>/dev/null || echo not) -ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/x86/ -e s/x86_64/x86/) +ARCH ?= $(shell uname -m 2>/dev/null || echo not) +override ARCH := $(shell echo $(ARCH) | sed -e s/i.86/x86/ -e s/x86_64/x86/) ifeq ($(ARCH),x86) TEST_PROGS := disable-tsc-ctxt-sw-stress-test disable-tsc-on-off-stress-test \ diff --git a/tools/testing/selftests/sparc64/Makefile b/tools/testing/selftests/sparc64/Makefile index a19531dba4dc..88f7be76f962 100644 --- a/tools/testing/selftests/sparc64/Makefile +++ b/tools/testing/selftests/sparc64/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 -uname_M := $(shell uname -m 2>/dev/null || echo not) -ARCH ?= $(shell echo $(uname_M) | sed -e s/x86_64/x86/) +ARCH ?= $(shell uname -m 2>/dev/null || echo not) +override ARCH := $(shell echo $(ARCH) | sed -e s/x86_64/x86/) ifneq ($(ARCH),sparc64) nothing: diff --git a/tools/testing/selftests/thermal/intel/power_floor/Makefile b/tools/testing/selftests/thermal/intel/power_floor/Makefile index 9b88e57dbba5..07463c2160e0 100644 --- a/tools/testing/selftests/thermal/intel/power_floor/Makefile +++ b/tools/testing/selftests/thermal/intel/power_floor/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 ifndef CROSS_COMPILE -uname_M := $(shell uname -m 2>/dev/null || echo not) -ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/x86/ -e s/x86_64/x86/) +ARCH ?= $(shell uname -m 2>/dev/null || echo not) +override ARCH := $(shell echo $(ARCH) | sed -e s/i.86/x86/ -e s/x86_64/x86/) ifeq ($(ARCH),x86) TEST_GEN_PROGS := power_floor_test diff --git a/tools/testing/selftests/thermal/intel/workload_hint/Makefile b/tools/testing/selftests/thermal/intel/workload_hint/Makefile index 37ff3286283b..49e3bc2b20f9 100644 --- a/tools/testing/selftests/thermal/intel/workload_hint/Makefile +++ b/tools/testing/selftests/thermal/intel/workload_hint/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 ifndef CROSS_COMPILE -uname_M := $(shell uname -m 2>/dev/null || echo not) -ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/x86/ -e s/x86_64/x86/) +ARCH ?= $(shell uname -m 2>/dev/null || echo not) +override ARCH := $(shell echo $(ARCH) | sed -e s/i.86/x86/ -e s/x86_64/x86/) ifeq ($(ARCH),x86) TEST_GEN_PROGS := workload_hint_test