Pull ntfs resurrection from Namjae Jeon:
"Ever since Kari Argillander’s 2022 report [1] regarding the state of
the ntfs3 driver, I have spent the last 4 years working to provide
full write support and current trends (iomap, no buffer head, folio),
enhanced performance, stable maintenance, utility support including
fsck for NTFS in Linux.
This new implementation is built upon the clean foundation of the
original read-only NTFS driver, adding:
- Write support:
Implemented full write support based on the classic read-only NTFS
driver. Added delayed allocation to improve write performance
through multi-cluster allocation and reduced fragmentation of the
cluster bitmap.
- iomap conversion:
Switched buffered IO (reads/writes), direct IO, file extent
mapping, readpages, and writepages to use iomap.
- Remove buffer_head:
Completely removed buffer_head usage by converting to folios. As a
result, the dependency on CONFIG_BUFFER_HEAD has been removed from
Kconfig.
- Stability improvements:
The new ntfs driver passes 326 xfstests, compared to 273 for ntfs3.
All tests passed by ntfs3 are a complete subset of the tests passed
by this implementation. Added support for fallocate, idmapped
mounts, permissions, and more.
xfstests Results report:
Total tests run: 787
Passed : 326
Failed : 38
Skipped : 423
Failed tests breakdown:
- 34 tests require metadata journaling
- 4 other tests:
094: No unwritten extent concept in NTFS on-disk format
563: cgroup v2 aware writeback accounting not supported
631: RENAME_WHITEOUT support required
787: NFS delegation test"
Link: https://lore.kernel.org/all/da20d32b-5185-f40b-48b8-2986922d8b25@stargateuniverse.net/ [1]
[ Let's see if this undead filesystem ends up being of the "Easter
miracle" kind, or the "Nosferatu of filesystems" kind... ]
* tag 'ntfs-for-7.1-rc1-v2' of git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/ntfs: (46 commits)
ntfs: remove redundant out-of-bound checks
ntfs: add bound checking to ntfs_external_attr_find
ntfs: add bound checking to ntfs_attr_find
ntfs: fix ignoring unreachable code warnings
ntfs: fix inconsistent indenting warnings
ntfs: fix variable dereferenced before check warnings
ntfs: prefer IS_ERR_OR_NULL() over manual NULL check
ntfs: harden ntfs_listxattr against EA entries
ntfs: harden ntfs_ea_lookup against malformed EA entries
ntfs: check $EA query-length in ntfs_ea_get
ntfs: validate WSL EA payload sizes
ntfs: fix WSL ea restore condition
ntfs: add missing newlines to pr_err() messages
ntfs: fix pointer/integer casting warnings
ntfs: use ->mft_no instead of ->i_ino in prints
ntfs: change mft_no type to u64
ntfs: select FS_IOMAP in Kconfig
ntfs: add MODULE_ALIAS_FS
ntfs: reduce stack usage in ntfs_write_mft_block()
ntfs: fix sysctl table registration and path
...
This improves log accuracy for NTFS debugging and removes unnecessary
reliance on the VFS i_ino field ahead of the core VFS type change.
Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>
Changes the type of ntfs_inode::mft_no from unsigned long to u64
to safely handle the full 48-bit range without truncation risk, especially
in preparation for broader VFS inode number type (i_ino:u64) and to
improve consistency with ntfs driver practices.
Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>
The use of two large arrays in this function makes the stack frame exceed
the warning limit in some configurations, especially with KASAN enabled.
When CONFIG_PAGE_SIZE is set to 65536, each of the arrays contains 128
pointers, so the combined size is 2KB:
fs/ntfs/mft.c: In function 'ntfs_write_mft_block.isra':
fs/ntfs/mft.c:2891:1: error: the frame size of 2640 bytes is larger than 1536 bytes [-Werror=frame-larger-than=]
Use dynamic allocation of these arrays to avoid getting into dangerously
high stack usage.
Unfortunately, allocating memory in the writepages() code path can be
problematic in case of low memory situations, so it would be better to
rework the code more widely to avoid the allocation entirely.
Fixes: 115380f9a2 ("ntfs: update mft operations")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>
Refactors MFT record handling to use folio APIs with consistency
validation, and improving allocation extension and writeback paths
for and .
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hyunchul Lee <hyc.lee@gmail.com>
Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>
This reverts commit 7ffa8f3d30.
Reverts the removal of the classic read-only ntfs driver to
serve as the base for a new read-write ntfs implementation.
If we stack changes on top of the revert patch, It will significantly
reduce the diff size, making the review easier.
This revert intentionally excludes the restoration of Kconfig and
Makefile. The Kconfig and Makefile will be added back in the final patch
of this series, enabling the driver only after all features and
improvements have been applied.
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>
In ntfs_mft_data_extend_allocation_nolock(), if an error condition occurs
prior to 'ctx' being set to a non-NULL value, avoid dereferencing the NULL
'ctx' pointer in error handling.
Found by Linux Verification Center (linuxtesting.org) with SVACE.
Signed-off-by: Danila Chernetsov <listdansp@mail.ru>
Reviewed-by: Namjae Jeon <linkinjeon@kernel.org>
Signed-off-by: Christian Brauner <brauner@kernel.org>
Fix many W=1 kernel-doc warnings in fs/ntfs/:
fs/ntfs/aops.c:30: warning: Incorrect use of kernel-doc format: * ntfs_end_buffer_async_read - async io completion for reading attributes
fs/ntfs/aops.c:46: warning: expecting prototype for aops.c(). Prototype was for ntfs_end_buffer_async_read() instead
fs/ntfs/aops.c:1655: warning: cannot understand function prototype: 'const struct address_space_operations ntfs_normal_aops = '
fs/ntfs/aops.c:1670: warning: cannot understand function prototype: 'const struct address_space_operations ntfs_compressed_aops = '
fs/ntfs/aops.c:1685: warning: cannot understand function prototype: 'const struct address_space_operations ntfs_mst_aops = '
fs/ntfs/compress.c:22: warning: Incorrect use of kernel-doc format: * ntfs_compression_constants - enum of constants used in the compression code
fs/ntfs/compress.c:24: warning: cannot understand function prototype: 'typedef enum '
fs/ntfs/compress.c:47: warning: cannot understand function prototype: 'u8 *ntfs_compression_buffer; '
fs/ntfs/compress.c:52: warning: expecting prototype for ntfs_cb_lock(). Prototype was for DEFINE_SPINLOCK() instead
fs/ntfs/dir.c:21: warning: Incorrect use of kernel-doc format: * The little endian Unicode string $I30 as a global constant.
fs/ntfs/dir.c:23: warning: cannot understand function prototype: 'ntfschar I30[5] = '
fs/ntfs/inode.c:31: warning: Incorrect use of kernel-doc format: * ntfs_test_inode - compare two (possibly fake) inodes for equality
fs/ntfs/inode.c:47: warning: expecting prototype for inode.c(). Prototype was for ntfs_test_inode() instead
fs/ntfs/inode.c:2956: warning: expecting prototype for ntfs_write_inode(). Prototype was for __ntfs_write_inode() instead
fs/ntfs/mft.c:24: warning: expecting prototype for mft.c - NTFS kernel mft record operations. Part of the Linux(). Prototype was for MAX_BHS() instead
fs/ntfs/namei.c:263: warning: This comment starts with '/**', but isn't a kernel-doc comment. Refer Documentation/doc-guide/kernel-doc.rst
* Inode operations for directories.
fs/ntfs/namei.c:368: warning: This comment starts with '/**', but isn't a kernel-doc comment. Refer Documentation/doc-guide/kernel-doc.rst
* Export operations allowing NFS exporting of mounted NTFS partitions.
fs/ntfs/runlist.c:16: warning: Incorrect use of kernel-doc format: * ntfs_rl_mm - runlist memmove
fs/ntfs/runlist.c:22: warning: expecting prototype for runlist.c - NTFS runlist handling code. Part of the Linux(). Prototype was for ntfs_rl_mm() instead
fs/ntfs/super.c:61: warning: missing initial short description on line:
* simple_getbool -
fs/ntfs/super.c:2661: warning: This comment starts with '/**', but isn't a kernel-doc comment. Refer Documentation/doc-guide/kernel-doc.rst
* The complete super operations.
Link: https://lkml.kernel.org/r/20230109010041.21442-1-rdunlap@infradead.org
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Cc: Anton Altaparmakov <anton@tuxera.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Clang's Control Flow Integrity (CFI) is a security mechanism that can help
prevent JOP chains, deployed extensively in downstream kernels used in
Android.
Its deployment is hindered by mismatches in function signatures. For this
case, we make callbacks match their intended function signature, and cast
parameters within them rather than casting the callback when passed as a
parameter.
When running `mount -t ntfs ...` we observe the following trace:
Call trace:
__cfi_check_fail+0x1c/0x24
name_to_dev_t+0x0/0x404
iget5_locked+0x594/0x5e8
ntfs_fill_super+0xbfc/0x43ec
mount_bdev+0x30c/0x3cc
ntfs_mount+0x18/0x24
mount_fs+0x1b0/0x380
vfs_kern_mount+0x90/0x398
do_mount+0x5d8/0x1a10
SyS_mount+0x108/0x144
el0_svc_naked+0x34/0x38
Signed-off-by: Luca Stefani <luca.stefani.ge1@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Tested-by: freak07 <michalechner92@googlemail.com>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Nathan Chancellor <natechancellor@gmail.com>
Acked-by: Anton Altaparmakov <anton@tuxera.com>
Link: http://lkml.kernel.org/r/20200718112513.533800-1-luca.stefani.ge1@gmail.com
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Based on 1 normalized pattern(s):
this program include file is free software you can redistribute it
and or modify it under the terms of the gnu general public license
as published by the free software foundation either version 2 of the
license or at your option any later version this program include
file is distributed in the hope that it will be useful but without
any warranty without even the implied warranty of merchantability or
fitness for a particular purpose see the gnu general public license
for more details you should have received a copy of the gnu general
public license along with this program in the main directory of the
linux [ntfs] distribution in the file copying if not write to the
free software foundation inc 59 temple place suite 330 boston ma
02111 1307 usa
extracted by the scancode license scanner the SPDX license identifier
GPL-2.0-or-later
has been chosen to replace the boilerplate/reference in 43 file(s).
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Richard Fontana <rfontana@redhat.com>
Reviewed-by: Allison Randal <allison@lohutok.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190520075212.517001706@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
I_DIRTY_DATASYNC is a strict superset of I_DIRTY_SYNC semantics, as
in mark dirty to be written out by fdatasync as well. So dirtying
for both flags makes no sense.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
NTFS keeps track of the i_version counter here, seemingly for no reason.
It does not set the SB_I_VERSION flag so it'll never be incremented on
write, and it doesn't increment it internally for metadata operations.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
Instead of adding weird retry logic in that function, utilize
__GFP_NOFAIL to ensure that the vm takes care of handling any
potential retries appropriately. This means we don't have to
call free_more_memory() from here.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
It's only needed for the CONFIG_SWAP-only use of bio_end_io_t.
Because CONFIG_SWAP implies CONFIG_BLOCK this will allow to drop some
ifdefs in blk_types.h.
Instead we'll need to add a few explicit includes that were implicit
before, though.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@fb.com>
current_fs_time() uses struct super_block* as an argument.
As per Linus's suggestion, this is changed to take struct
inode* as a parameter instead. This is because the function
is primarily meant for vfs inode timestamps.
Also the function was renamed as per Arnd's suggestion.
Change all calls to current_fs_time() to use the new
current_time() function instead. current_fs_time() will be
deleted.
Signed-off-by: Deepa Dinamani <deepa.kernel@gmail.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
This has submit_bh users pass in the operation and flags separately,
so submit_bh_wbc can setup the bio op and bi_rw flags on the bio that
is submitted.
Signed-off-by: Mike Christie <mchristi@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
In ntfs_mft_record_alloc() when mapping the new extent mft record with
map_extent_mft_record() we overwrite @m with the return value and on
error, we then try to use the old @m but that is no longer there as @m
now contains an error code instead so we crash when dereferencing the
error code as if it were a pointer.
The simple fix is to use a temporary variable to store the return value
thus preserving the original @m for later use. This is a backport from
the commercial Tuxera-NTFS driver and is well tested...
Thanks go to Julia Lawall for pointing this out (whilst I had fixed it
in the commercial driver I had failed to fix it in the Linux kernel).
Signed-off-by: Anton Altaparmakov <anton@tuxera.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
Use new syncing helpers in .write and .aio_write functions. Also
remove superfluous syncing in ntfs_file_buffered_write() and update
comments about generic_osync_inode().
CC: Anton Altaparmakov <aia21@cantab.net>
CC: linux-ntfs-dev@lists.sourceforge.net
Signed-off-by: Jan Kara <jack@suse.cz>
Like the page lock change, this also requires name change, so convert the
raw test_and_set bitop to a trylock.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
This eliminates the i_blksize field from struct inode. Filesystems that want
to provide a per-inode st_blksize can do so by providing their own getattr
routine instead of using the generic_fillattr() function.
Note that some filesystems were providing pretty much random (and incorrect)
values for i_blksize.
[bunk@stusta.de: cleanup]
[akpm@osdl.org: generic_fillattr() fix]
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Now the only user who are using generic_ffs() is ntfs filesystem. This patch
isolates generic_ffs() as ntfs_ffs() for ntfs.
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>
Cc: Anton Altaparmakov <aia21@cantab.net>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
The conversion was generated via scripts, and the result was validated
automatically via a script as well.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Anton Altaparmakov <aia21@cantab.net>
forgot to update a temporary variable so loading index inodes which
have an index allocation attribute failed.
Signed-off-by: Anton Altaparmakov <aia21@cantab.net>
specifying whether the cluster are being allocated to extend an
attribute or to fill a hole.
- Change ntfs_attr_make_non_resident() to call ntfs_cluster_alloc()
with @is_extension set to TRUE and remove the runlist terminator
fixup code as this is now done by ntfs_cluster_alloc().
Signed-off-by: Anton Altaparmakov <aia21@cantab.net>
search context as argument. This allows calling it with the mft
record mapped. Update all callers.
- Fix potential deadlock in ntfs_mft_data_extend_allocation_nolock()
error handling by passing in the active search context when calling
ntfs_cluster_free().
Signed-off-by: Anton Altaparmakov <aia21@cantab.net>
search context as argument. This allows calling it with the mft
record mapped. Update all callers.
Signed-off-by: Anton Altaparmakov <aia21@cantab.net>
since we otherwise get into a lock reversal deadlock if a read locked
runlist is passed in. In the process also change it to take an ntfs
inode instead of a vfs inode as parameter.
Signed-off-by: Anton Altaparmakov <aia21@cantab.net>
the buffers when mapping them after the VM had discarded them.
Thanks to Martin MOKREJŠ for the bug report.
Signed-off-by: Anton Altaparmakov <aia21@cantab.net>
