Discussion:
Error setting extent size on a directory
Iustin Pop
2014-07-14 07:09:13 UTC
Permalink
Hi,

During the restore of a file system, a few directories failed to have
their extent size set:

xfsrestore: WARNING: attempt to set extended attributes (xflags 0x1080,
extsize = 0x800000, projid = 0x0) of a failed: Invalid argument

Since there were just a few, I've set the nodump flag (0x80 in flags
above) manually, which worked, but trying to set the extent size still
fails:
xfs_io .
xfs_io> extsize
[0] .
xfs_io> extsize 8m
xfs_io: XFS_IOC_FSSETXATTR .: Invalid argument

The xfsctl man page says that an extent size should be settable any time
on a directory, so why would this fail? Looking at the kernel sources,
I see a number of possible cases where EINVAL is returned:

- ip->i_d.di_nextents non-zero
- requested extend size bigger than MAXEXTLEN; not the case here
- extsize_fsb > mp->m_sb.sb_agblocks / 2; not the case
- fa->fsx_extsize % size - extent size not a multiple of FS block size;
again not the case

So to me this reads as if the di_nextents check can also fail for a
directory which has extents, contradicting the man page. Which one needs
to be updated?

The question arises to if the extent size also applies, then, to
allocating extents for a directory - instead of just being inherited for
files (the man page says no).

thanks,
iustin
Christoph Hellwig
2014-07-17 09:04:50 UTC
Permalink
Post by Iustin Pop
The xfsctl man page says that an extent size should be settable any time
on a directory, so why would this fail? Looking at the kernel sources,
And no special casing for directories at all..
Post by Iustin Pop
So to me this reads as if the di_nextents check can also fail for a
directory which has extents, contradicting the man page. Which one needs
to be updated?
The question arises to if the extent size also applies, then, to
allocating extents for a directory - instead of just being inherited for
files (the man page says no).
We're not using the extent size hint on the directory itself. So to
me it seems we just not check for already allocated blocks if we're
setting the extent size on a directory, but instead maybe make sure
the directory. What's also a little odd is that we allow setting
the extent size on a directory even if the extent size inherit bit is
not set, which doesn't make much sense to me.

Do you want to prepare a patch to remove the check for directories?
At testcase for xfstests that ensures this works also would be highly
useful..
Iustin Pop
2014-07-18 19:13:14 UTC
Permalink
Post by Christoph Hellwig
Post by Iustin Pop
The xfsctl man page says that an extent size should be settable any time
on a directory, so why would this fail? Looking at the kernel sources,
And no special casing for directories at all..
I was not sure if di_nextents is valid (tracks the extents number, and
hence can be non-zero for a directory), I'll take this as a
confirmation.
Post by Christoph Hellwig
Post by Iustin Pop
So to me this reads as if the di_nextents check can also fail for a
directory which has extents, contradicting the man page. Which one needs
to be updated?
The question arises to if the extent size also applies, then, to
allocating extents for a directory - instead of just being inherited for
files (the man page says no).
We're not using the extent size hint on the directory itself.
Aha, this is good to know.
Post by Christoph Hellwig
So to
me it seems we just not check for already allocated blocks if we're
setting the extent size on a directory, but instead maybe make sure
the directory.
Not sure I parse that - do you mean we should either check for a
directory, or for zero extent count?
Post by Christoph Hellwig
What's also a little odd is that we allow setting
the extent size on a directory even if the extent size inherit bit is
not set, which doesn't make much sense to me.
Since the hint it is never used for directories, agreed it doesn't make
sense. Should this be an error (since I don't think warnings can be
reported).
Post by Christoph Hellwig
Do you want to prepare a patch to remove the check for directories?
At testcase for xfstests that ensures this works also would be highly
useful..
I'll try to. Is the tree against which I should sent the kernel patch at
git://oss.sgi.com/xfs/xfs?

thanks,
iustin
Iustin Pop
2014-08-28 04:22:53 UTC
Permalink
Currently, the ioctl handling code for XFS_IOC_FSSETXATTR treats all
targets as regular files: it refuses to change the extent size if
extents are allocated. This is wrong for directories, as there the
extent size is only used as a default for children.

The patch fixes this by only checking for allocated extents for regular
files; it leaves undefined what it means to set an extent size on a
non-regular, non-directory inode. Additionally, when setting a non-zero
extent size, the appropriate flags (EXTSIZE, respectively EXTSZINHERIT)
are enforced for regular files and directories.

Signed-off-by: Iustin Pop <***@k1024.org>
---
A patch against xfstests to test for the fixed behaviour will follow
shortly.

fs/xfs/xfs_ioctl.c | 34 ++++++++++++++++++++++++++--------
1 file changed, 26 insertions(+), 8 deletions(-)

diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c
index 8bc1bbc..5b9acd2 100644
--- a/fs/xfs/xfs_ioctl.c
+++ b/fs/xfs/xfs_ioctl.c
@@ -1116,14 +1116,32 @@ xfs_ioctl_setattr(
}

if (mask & FSX_EXTSIZE) {
- /*
- * Can't change extent size if any extents are allocated.
- */
- if (ip->i_d.di_nextents &&
- ((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) !=
- fa->fsx_extsize)) {
- code = XFS_ERROR(EINVAL); /* EFBIG? */
- goto error_return;
+ if (S_ISDIR(ip->i_d.di_mode)) {
+ /*
+ * Enforce setting the EXTSZINHERIT flag when
+ * a non-zero extent size has been specified.
+ */
+ if (fa->fsx_extsize) {
+ fa->fsx_xflags |= XFS_XFLAG_EXTSZINHERIT;
+ }
+ } else if (S_ISREG(ip->i_d.di_mode)) {
+ /*
+ * For a regular file, we can't change extent
+ * size if any extents are allocated.
+ */
+ if (ip->i_d.di_nextents &&
+ ((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) !=
+ fa->fsx_extsize)) {
+ code = XFS_ERROR(EINVAL); /* EFBIG? */
+ goto error_return;
+ }
+ /*
+ * Enforce setting the EXTSIZE flag when
+ * a non-zero extent size has been specified.
+ */
+ if (fa->fsx_extsize) {
+ fa->fsx_xflags |= XFS_XFLAG_EXTSIZE;
+ }
}

/*
--
2.1.0
Dave Chinner
2014-08-28 09:31:58 UTC
Permalink
Post by Iustin Pop
Currently, the ioctl handling code for XFS_IOC_FSSETXATTR treats all
targets as regular files: it refuses to change the extent size if
extents are allocated. This is wrong for directories, as there the
extent size is only used as a default for children.
The patch fixes this by only checking for allocated extents for regular
files; it leaves undefined what it means to set an extent size on a
non-regular, non-directory inode. Additionally, when setting a non-zero
extent size, the appropriate flags (EXTSIZE, respectively EXTSZINHERIT)
are enforced for regular files and directories.
---
A patch against xfstests to test for the fixed behaviour will follow
shortly.
fs/xfs/xfs_ioctl.c | 34 ++++++++++++++++++++++++++--------
1 file changed, 26 insertions(+), 8 deletions(-)
diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c
index 8bc1bbc..5b9acd2 100644
--- a/fs/xfs/xfs_ioctl.c
+++ b/fs/xfs/xfs_ioctl.c
@@ -1116,14 +1116,32 @@ xfs_ioctl_setattr(
}
if (mask & FSX_EXTSIZE) {
- /*
- * Can't change extent size if any extents are allocated.
- */
- if (ip->i_d.di_nextents &&
- ((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) !=
- fa->fsx_extsize)) {
- code = XFS_ERROR(EINVAL); /* EFBIG? */
- goto error_return;
Doesn't apply to a current 3.17-rc1 tree. Can you update the patch?
Post by Iustin Pop
+ if (S_ISDIR(ip->i_d.di_mode)) {
+ /*
+ * Enforce setting the EXTSZINHERIT flag when
+ * a non-zero extent size has been specified.
+ */
+ if (fa->fsx_extsize) {
+ fa->fsx_xflags |= XFS_XFLAG_EXTSZINHERIT;
+ }
+ } else if (S_ISREG(ip->i_d.di_mode)) {
+ /*
+ * For a regular file, we can't change extent
+ * size if any extents are allocated.
+ */
+ if (ip->i_d.di_nextents &&
+ ((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) !=
+ fa->fsx_extsize)) {
+ code = XFS_ERROR(EINVAL); /* EFBIG? */
+ goto error_return;
+ }
+ /*
+ * Enforce setting the EXTSIZE flag when
+ * a non-zero extent size has been specified.
+ */
+ if (fa->fsx_extsize) {
+ fa->fsx_xflags |= XFS_XFLAG_EXTSIZE;
+ }
}
Hmmmm. That's not validating/enforcing the correct use of
XFS_XFLAG_EXTSIZE or XFS_XFLAG_EXTSZINHERIT, that's setting it
implicitly based on the type of inode.

If we are going to enforce this properly, then XFS_XFLAG_EXTSIZE is
only valid for a regular file, and XFS_XFLAG_EXTSZINHERIT is only
valid on a directory, and the flags on th einode should only be set
if the hint is not zero. i.e:

if (mask & FSX_EXTSIZE) {
error = -EINVAL;

/* validate the flags are set appropriately */
if ((fa->fsx_xflags & XFS_XFLAG_EXTSIZE) &&
!S_ISREG(ip->i_d.di_mode))
goto error_return;
if ((fa->fsx_xflags & XFS_XFLAG_EXTSZINHERIT) &&
!S_ISDIR(ip->i_d.di_mode))
goto error_return;

/* Can't change extent size on regular files with allocated extents */
if (S_ISREG(ip->i_d.di_mode) && ip->i_d.di_nextents &&
((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) !=
fa->fsx_extsize))
goto error_return;

/* if the extent size is zero, clear the inode flags */
if (fs->fsx_extsize == 0)
fa->fsx_xflags &= ~(XFS_XFLAG_EXTSIZE | XFS_XFLAG_EXTSZINHERIT);
} else {
/* existing size check code */
....
}
}

Cheers,

Dave.
--
Dave Chinner
***@fromorbit.com
Iustin Pop
2014-08-28 22:34:14 UTC
Permalink
Post by Dave Chinner
Post by Iustin Pop
Currently, the ioctl handling code for XFS_IOC_FSSETXATTR treats all
targets as regular files: it refuses to change the extent size if
extents are allocated. This is wrong for directories, as there the
extent size is only used as a default for children.
The patch fixes this by only checking for allocated extents for regular
files; it leaves undefined what it means to set an extent size on a
non-regular, non-directory inode. Additionally, when setting a non-zero
extent size, the appropriate flags (EXTSIZE, respectively EXTSZINHERIT)
are enforced for regular files and directories.
---
A patch against xfstests to test for the fixed behaviour will follow
shortly.
fs/xfs/xfs_ioctl.c | 34 ++++++++++++++++++++++++++--------
1 file changed, 26 insertions(+), 8 deletions(-)
diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c
index 8bc1bbc..5b9acd2 100644
--- a/fs/xfs/xfs_ioctl.c
+++ b/fs/xfs/xfs_ioctl.c
@@ -1116,14 +1116,32 @@ xfs_ioctl_setattr(
}
if (mask & FSX_EXTSIZE) {
- /*
- * Can't change extent size if any extents are allocated.
- */
- if (ip->i_d.di_nextents &&
- ((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) !=
- fa->fsx_extsize)) {
- code = XFS_ERROR(EINVAL); /* EFBIG? */
- goto error_return;
Doesn't apply to a current 3.17-rc1 tree. Can you update the patch?
Will do, thanks. Is it correct to use git://oss.sgi.com/xfs/xfs, master
branch as what I should rebase it on top of?
Post by Dave Chinner
Post by Iustin Pop
+ if (S_ISDIR(ip->i_d.di_mode)) {
+ /*
+ * Enforce setting the EXTSZINHERIT flag when
+ * a non-zero extent size has been specified.
+ */
+ if (fa->fsx_extsize) {
+ fa->fsx_xflags |= XFS_XFLAG_EXTSZINHERIT;
+ }
+ } else if (S_ISREG(ip->i_d.di_mode)) {
+ /*
+ * For a regular file, we can't change extent
+ * size if any extents are allocated.
+ */
+ if (ip->i_d.di_nextents &&
+ ((ip->i_d.di_extsize << mp->m_sb.sb_blocklog) !=
+ fa->fsx_extsize)) {
+ code = XFS_ERROR(EINVAL); /* EFBIG? */
+ goto error_return;
+ }
+ /*
+ * Enforce setting the EXTSIZE flag when
+ * a non-zero extent size has been specified.
+ */
+ if (fa->fsx_extsize) {
+ fa->fsx_xflags |= XFS_XFLAG_EXTSIZE;
+ }
}
Hmmmm. That's not validating/enforcing the correct use of
XFS_XFLAG_EXTSIZE or XFS_XFLAG_EXTSZINHERIT, that's setting it
implicitly based on the type of inode.
If we are going to enforce this properly, then XFS_XFLAG_EXTSIZE is
only valid for a regular file, and XFS_XFLAG_EXTSZINHERIT is only
valid on a directory, and the flags on th einode should only be set
if (mask & FSX_EXTSIZE) {
error = -EINVAL;
/* validate the flags are set appropriately */
if ((fa->fsx_xflags & XFS_XFLAG_EXTSIZE) &&
!S_ISREG(ip->i_d.di_mode))
goto error_return;
if ((fa->fsx_xflags & XFS_XFLAG_EXTSZINHERIT) &&
!S_ISDIR(ip->i_d.di_mode))
goto error_return;
This only prevents invalid flags, but doesn't prevent setting a non-zero
extent_size without the appropriate flag. We already are having this
discussion in the other patch, so I'll defer this to that.

But aside from that, the current code that actually applies the flag
changes (xfs_set_diflags) implicitly drops invalid flags - that's why I
didn't bother to validate the flags here. If we want to do validation,
maybe we should validate (with error, not silently) in that function,
instead of here?

thanks,
iustin
Dave Chinner
2014-08-29 00:46:07 UTC
Permalink
This post might be inappropriate. Click to display it.
Iustin Pop
2014-08-28 04:24:00 UTC
Permalink
Add two tests that check for correct behaviour of XFS_IOC_FSSETXATTR:

- 307: check that extent size can always be set on a directory
- 308: check that setting a non-zero extent size directly via the ioctl
sets the expected flags (EXTSIZE and EXTSZINHERIT)

Signed-off-by: Iustin Pop <***@k1024.org>
---
tests/xfs/307 | 87 ++++++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/307.out | 3 ++
tests/xfs/308 | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/308.out | 3 ++
tests/xfs/group | 2 ++
5 files changed, 193 insertions(+)
create mode 100755 tests/xfs/307
create mode 100644 tests/xfs/307.out
create mode 100755 tests/xfs/308
create mode 100644 tests/xfs/308.out

diff --git a/tests/xfs/307 b/tests/xfs/307
new file mode 100755
index 0000000..e8f3576
--- /dev/null
+++ b/tests/xfs/307
@@ -0,0 +1,87 @@
+#! /bin/bash
+# FS QA Test No. 307
+#
+# Test that setting extent size on directories work even for large
+# directories.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2014 Google Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs xfs
+_supported_os Linux
+_require_test
+_require_scratch
+
+_scratch_mkfs_xfs >/dev/null 2>&1
+_scratch_mount
+
+small=$SCRATCH_MNT/small
+big=$SCRATCH_MNT/big
+
+# sanity check on a small directory
+mkdir $small
+# expect that an empty directory has no extents
+xfs_bmap $small | grep -q "no extents"
+# and that we can set an extent size on it
+xfs_io -c 'extsize 8m' $small
+# and finally check that the extent size update has taken place
+(cd $SCRATCH_MNT; xfs_io -c 'extsize' small)
+
+# now create a 'big' (with extents) directory
+mkdir $big
+idx=1
+while xfs_bmap $big | grep -q "no extents"; do
+ touch $big/$idx
+ idx=$((idx+1))
+ if [ "$idx" -gt 1048576 ]; then
+ # still no extents? giving up
+ echo "Can't make a directory to have extents even after 1M files" 1>&2
+ exit
+ fi
+done
+
+# expect that we can set the extent size on $big as well
+xfs_io -c 'extsize 8m' $big
+# and that it took effect
+(cd $SCRATCH_MNT; xfs_io -c 'extsize' big)
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/307.out b/tests/xfs/307.out
new file mode 100644
index 0000000..f825525
--- /dev/null
+++ b/tests/xfs/307.out
@@ -0,0 +1,3 @@
+QA output created by 307
+[8388608] small
+[8388608] big
diff --git a/tests/xfs/308 b/tests/xfs/308
new file mode 100755
index 0000000..7b43836
--- /dev/null
+++ b/tests/xfs/308
@@ -0,0 +1,98 @@
+#! /bin/bash
+# FS QA Test No. 308
+#
+# Test that setting extent size on files and directories (directly via
+# ioctl and not via xfs_io) sets the correct flags.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2014 Google Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs xfs
+_supported_os Linux
+_require_test
+_require_scratch
+
+_scratch_mkfs_xfs >/dev/null 2>&1
+_scratch_mount
+
+file=$SCRATCH_MNT/file
+dir=$SCRATCH_MNT/dir
+cprog=$tmp.get_structs.c
+oprog=$tmp.get_structs
+
+touch $file
+mkdir $dir
+
+cat > $cprog << EOF
+#include <stdio.h>
+#include <xfs/xfs.h>
+
+int main(int argc, char **argv) {
+ struct fsxattr fa;
+ int fd = open(argv[1], O_RDONLY);
+ if (fd < 0) {
+ perror("open");
+ return 1;
+ }
+ fa.fsx_xflags = 0;
+ fa.fsx_extsize = 1048576 * 8;
+ int r = xfsctl(argv[1], fd, XFS_IOC_FSSETXATTR, &fa);
+ if (r < 0) {
+ perror("xfsctl");
+ return 1;
+ }
+ return 0;
+}
+EOF
+cc -o $oprog $cprog >> $seqres.full 2>&1 || \
+ _notrun "Could not compile test program (see end of $seqres.full)"
+
+$oprog $file
+$oprog $dir
+(cd $SCRATCH_MNT;
+ xfs_io -c 'lsattr' file;
+ xfs_io -c 'lsattr' dir)
+
+rm $file
+rmdir $dir
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/308.out b/tests/xfs/308.out
new file mode 100644
index 0000000..ef2d5fd
--- /dev/null
+++ b/tests/xfs/308.out
@@ -0,0 +1,3 @@
+QA output created by 308
+----------e--- file
+-----------E-- dir
diff --git a/tests/xfs/group b/tests/xfs/group
index 4d35df5..35f1e6a 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -197,3 +197,5 @@
304 auto quick quota
305 auto quota
306 auto stress log metadata repair
+307 ioctl quick
+308 ioctl quick
--
2.1.0
Dave Chinner
2014-08-28 10:16:28 UTC
Permalink
Post by Iustin Pop
- 307: check that extent size can always be set on a directory
- 308: check that setting a non-zero extent size directly via the ioctl
sets the expected flags (EXTSIZE and EXTSZINHERIT)
Minor stuff first:

- xfstests patches should be sent to ***@vger.kernel.org now.
- can you pick the first unused numbers in the sequence for new
tests (xfs/032, xfs/051) so I don't have to renumber them before
applying them?
- a patch per new test - it makes it easier to review and apply as i
don't have to split patches into multiple commits...
Post by Iustin Pop
diff --git a/tests/xfs/307 b/tests/xfs/307
new file mode 100755
index 0000000..e8f3576
--- /dev/null
+++ b/tests/xfs/307
@@ -0,0 +1,87 @@
+#! /bin/bash
+# FS QA Test No. 307
+#
+# Test that setting extent size on directories work even for large
+# directories.
What is a "large directory"? Wouldn't it be better to describe the
test as "Determine whether the extent size hint can be set on
directories with allocated extents correctly"?
Post by Iustin Pop
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2014 Google Inc. All Rights Reserved.
Is that correct? It doesn't match the email address you sent this
from and I've never seen you post from a @google.com address. I
always like to check that the copyright assignment is correct in
situations like this...
Post by Iustin Pop
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs xfs
+_supported_os Linux
+_require_test
Not needed, doesn't use the test device.
Post by Iustin Pop
+_require_scratch
+
+_scratch_mkfs_xfs >/dev/null 2>&1
+_scratch_mount
+
+small=$SCRATCH_MNT/small
+big=$SCRATCH_MNT/big
+
+# sanity check on a small directory
+mkdir $small
+# expect that an empty directory has no extents
+xfs_bmap $small | grep -q "no extents"
Better to use xfs_io directly and filter out $SCRATCH_MNT into the
golden output file like so:

$XFS_IO_PROG -c "bmap" $small | _filter_scratch

which will give:

SCRATCH_MNT/small: no extents
Post by Iustin Pop
+# and that we can set an extent size on it
+xfs_io -c 'extsize 8m' $small
$XFS_IO_PROG
Post by Iustin Pop
+# and finally check that the extent size update has taken place
+(cd $SCRATCH_MNT; xfs_io -c 'extsize' small)
$XFS_IO_PROG -c 'extsize' $small | _filter_scratch
Post by Iustin Pop
+# now create a 'big' (with extents) directory
+mkdir $big
+idx=1
+while xfs_bmap $big | grep -q "no extents"; do
+ touch $big/$idx
+ idx=$((idx+1))
+ if [ "$idx" -gt 1048576 ]; then
+ # still no extents? giving up
+ echo "Can't make a directory to have extents even after 1M files" 1>&2
+ exit
+ fi
+done
urk. largest inode size is 2kb, which means at most it can fit less
than 100 dirents in the inode before spilling to extent form. So
just do a loop that creates 1000 files - there's no need to
overengineer the test code.
Post by Iustin Pop
+# expect that we can set the extent size on $big as well
+xfs_io -c 'extsize 8m' $big
$XFS_IO_PROG
Post by Iustin Pop
+# and that it took effect
+(cd $SCRATCH_MNT; xfs_io -c 'extsize' big)
$XFS_IO_PROG -c 'extsize' $big | _filter_scratch
Post by Iustin Pop
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/307.out b/tests/xfs/307.out
new file mode 100644
index 0000000..f825525
--- /dev/null
+++ b/tests/xfs/307.out
@@ -0,0 +1,3 @@
+QA output created by 307
+[8388608] small
+[8388608] big
diff --git a/tests/xfs/308 b/tests/xfs/308
new file mode 100755
index 0000000..7b43836
--- /dev/null
+++ b/tests/xfs/308
@@ -0,0 +1,98 @@
+#! /bin/bash
+# FS QA Test No. 308
+#
+# Test that setting extent size on files and directories (directly via
+# ioctl and not via xfs_io) sets the correct flags.
xfs_io uses the ioctl directly - there's no need to write a special
c program to do this, especially as....
Post by Iustin Pop
+touch $file
+mkdir $dir
+
+cat > $cprog << EOF
+#include <stdio.h>
+#include <xfs/xfs.h>
+
+int main(int argc, char **argv) {
+ struct fsxattr fa;
+ int fd = open(argv[1], O_RDONLY);
+ if (fd < 0) {
+ perror("open");
+ return 1;
+ }
+ fa.fsx_xflags = 0;
+ fa.fsx_extsize = 1048576 * 8;
+ int r = xfsctl(argv[1], fd, XFS_IOC_FSSETXATTR, &fa);
.... that code is quite broken. Yes, it would work to set the
appropriate extent size flags with the kernel
changes you made, but that's not how this ioctl works.

i.e. it will cause any flag bits that are set on the inode to be
cleared and it's likely to fail on old kernels beacuse they have
very different behaviour to what your patch does.

IOWs, setting fsx_extsize without setting either XFS_XFLAG_EXTSIZE
or XFS_XFLAG_EXTSZINHERIT is bad practice. The kernel is left to
guess what you actually wanted to be done - the flags are supposed
to tell the kernel that the fsx_extsize value is meaningful, not the
other way around.

FWIW, the xfs_io code is *exactly* what applications should be
doing to set the extent size or any other inode flag. i.e:

1. stat the fd to determine the type.
2. populate the fsxattr structure with the existing inode flags
3. change the flags/fields of the fsxattr structure appropriately
4. set the new values to the inode.

i.e, from io/open.c:

static int
set_extsize(const char *path, int fd, long extsz)
{
struct fsxattr fsx;
struct stat64 stat;

if (fstat64(fd, &stat) < 0) {
perror("fstat64");
return 0;
}
if ((xfsctl(path, fd, XFS_IOC_FSGETXATTR, &fsx)) < 0) {
printf("%s: XFS_IOC_FSGETXATTR %s: %s\n",
progname, path, strerror(errno));
return 0;
}

if (S_ISREG(stat.st_mode)) {
fsx.fsx_xflags |= XFS_XFLAG_EXTSIZE;
} else if (S_ISDIR(stat.st_mode)) {
fsx.fsx_xflags |= XFS_XFLAG_EXTSZINHERIT;
} else {
printf(_("invalid target file type - file %s\n"), path);
return 0;
}
fsx.fsx_extsize = extsz;

if ((xfsctl(path, fd, XFS_IOC_FSSETXATTR, &fsx)) < 0) {
printf("%s: XFS_IOC_FSSETXATTR %s: %s\n",
progname, path, strerror(errno));
return 0;
}

return 0;
}

We have xfs_io precisely so that we don't have to maintain random
test code like this throughout xfstests - do it once, do it right,
use it everywhere.

Cheers,

Dave.
--
Dave Chinner
***@fromorbit.com
Iustin Pop
2014-08-28 22:28:54 UTC
Permalink
Post by Dave Chinner
Post by Iustin Pop
- 307: check that extent size can always be set on a directory
- 308: check that setting a non-zero extent size directly via the ioctl
sets the expected flags (EXTSIZE and EXTSZINHERIT)
OK, will do. There was nothing written in the git repository's README,
hence I chose what I thought best.
Post by Dave Chinner
- can you pick the first unused numbers in the sequence for new
tests (xfs/032, xfs/051) so I don't have to renumber them before
applying them?
Will do - this is what the 'new' script did (or tried to do, as it
doesn't seem to work reliably).
Post by Dave Chinner
- a patch per new test - it makes it easier to review and apply as i
don't have to split patches into multiple commits...
Will do so when resending.
Post by Dave Chinner
Post by Iustin Pop
diff --git a/tests/xfs/307 b/tests/xfs/307
new file mode 100755
index 0000000..e8f3576
--- /dev/null
+++ b/tests/xfs/307
@@ -0,0 +1,87 @@
+#! /bin/bash
+# FS QA Test No. 307
+#
+# Test that setting extent size on directories work even for large
+# directories.
What is a "large directory"? Wouldn't it be better to describe the
test as "Determine whether the extent size hint can be set on
directories with allocated extents correctly"?
Indeed that's better, I'll update.
Post by Dave Chinner
Post by Iustin Pop
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2014 Google Inc. All Rights Reserved.
Is that correct? It doesn't match the email address you sent this
always like to check that the copyright assignment is correct in
situations like this...
It is correct indeed (and thanks for double-checking). I prefer to send
my interactions/contributions done not as part of my job using my
personal address (hence I always wrote to xfs@ from the same address),
but even in that case, the copyright remains with my employer.
Post by Dave Chinner
Post by Iustin Pop
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs xfs
+_supported_os Linux
+_require_test
Not needed, doesn't use the test device.
Ack.
Post by Dave Chinner
Post by Iustin Pop
+_require_scratch
+
+_scratch_mkfs_xfs >/dev/null 2>&1
+_scratch_mount
+
+small=$SCRATCH_MNT/small
+big=$SCRATCH_MNT/big
+
+# sanity check on a small directory
+mkdir $small
+# expect that an empty directory has no extents
+xfs_bmap $small | grep -q "no extents"
Better to use xfs_io directly and filter out $SCRATCH_MNT into the
$XFS_IO_PROG -c "bmap" $small | _filter_scratch
SCRATCH_MNT/small: no extents
Oh, nice, thanks!
Post by Dave Chinner
Post by Iustin Pop
+# and that we can set an extent size on it
+xfs_io -c 'extsize 8m' $small
$XFS_IO_PROG
Ack.
Post by Dave Chinner
Post by Iustin Pop
+# and finally check that the extent size update has taken place
+(cd $SCRATCH_MNT; xfs_io -c 'extsize' small)
$XFS_IO_PROG -c 'extsize' $small | _filter_scratch
Ack.
Post by Dave Chinner
Post by Iustin Pop
+# now create a 'big' (with extents) directory
+mkdir $big
+idx=1
+while xfs_bmap $big | grep -q "no extents"; do
+ touch $big/$idx
+ idx=$((idx+1))
+ if [ "$idx" -gt 1048576 ]; then
+ # still no extents? giving up
+ echo "Can't make a directory to have extents even after 1M files" 1>&2
+ exit
+ fi
+done
urk. largest inode size is 2kb, which means at most it can fit less
than 100 dirents in the inode before spilling to extent form. So
just do a loop that creates 1000 files - there's no need to
overengineer the test code.
Will do. It's fine to still check that the directory does have extents,
I hope?
Post by Dave Chinner
Post by Iustin Pop
+# expect that we can set the extent size on $big as well
+xfs_io -c 'extsize 8m' $big
$XFS_IO_PROG
Ack.
Post by Dave Chinner
Post by Iustin Pop
+# and that it took effect
+(cd $SCRATCH_MNT; xfs_io -c 'extsize' big)
$XFS_IO_PROG -c 'extsize' $big | _filter_scratch
Ack.
Post by Dave Chinner
Post by Iustin Pop
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/307.out b/tests/xfs/307.out
new file mode 100644
index 0000000..f825525
--- /dev/null
+++ b/tests/xfs/307.out
@@ -0,0 +1,3 @@
+QA output created by 307
+[8388608] small
+[8388608] big
diff --git a/tests/xfs/308 b/tests/xfs/308
new file mode 100755
index 0000000..7b43836
--- /dev/null
+++ b/tests/xfs/308
@@ -0,0 +1,98 @@
+#! /bin/bash
+# FS QA Test No. 308
+#
+# Test that setting extent size on files and directories (directly via
+# ioctl and not via xfs_io) sets the correct flags.
xfs_io uses the ioctl directly - there's no need to write a special
c program to do this, especially as....
Post by Iustin Pop
+touch $file
+mkdir $dir
+
+cat > $cprog << EOF
+#include <stdio.h>
+#include <xfs/xfs.h>
+
+int main(int argc, char **argv) {
+ struct fsxattr fa;
+ int fd = open(argv[1], O_RDONLY);
+ if (fd < 0) {
+ perror("open");
+ return 1;
+ }
+ fa.fsx_xflags = 0;
+ fa.fsx_extsize = 1048576 * 8;
+ int r = xfsctl(argv[1], fd, XFS_IOC_FSSETXATTR, &fa);
.... that code is quite broken. Yes, it would work to set the
appropriate extent size flags with the kernel
changes you made, but that's not how this ioctl works.
i.e. it will cause any flag bits that are set on the inode to be
cleared
Good point…
Post by Dave Chinner
and it's likely to fail on old kernels beacuse they have
very different behaviour to what your patch does.
OK, that I didn't know. (Would you mind quickly explaining?)
Post by Dave Chinner
IOWs, setting fsx_extsize without setting either XFS_XFLAG_EXTSIZE
or XFS_XFLAG_EXTSZINHERIT is bad practice. The kernel is left to
guess what you actually wanted to be done - the flags are supposed
to tell the kernel that the fsx_extsize value is meaningful, not the
other way around.
See below.
Post by Dave Chinner
FWIW, the xfs_io code is *exactly* what applications should be
1. stat the fd to determine the type.
2. populate the fsxattr structure with the existing inode flags
3. change the flags/fields of the fsxattr structure appropriately
4. set the new values to the inode.
static int
set_extsize(const char *path, int fd, long extsz)
{
struct fsxattr fsx;
struct stat64 stat;
if (fstat64(fd, &stat) < 0) {
perror("fstat64");
return 0;
}
if ((xfsctl(path, fd, XFS_IOC_FSGETXATTR, &fsx)) < 0) {
printf("%s: XFS_IOC_FSGETXATTR %s: %s\n",
progname, path, strerror(errno));
return 0;
}
if (S_ISREG(stat.st_mode)) {
fsx.fsx_xflags |= XFS_XFLAG_EXTSIZE;
} else if (S_ISDIR(stat.st_mode)) {
fsx.fsx_xflags |= XFS_XFLAG_EXTSZINHERIT;
} else {
printf(_("invalid target file type - file %s\n"), path);
return 0;
}
fsx.fsx_extsize = extsz;
if ((xfsctl(path, fd, XFS_IOC_FSSETXATTR, &fsx)) < 0) {
printf("%s: XFS_IOC_FSSETXATTR %s: %s\n",
progname, path, strerror(errno));
return 0;
}
return 0;
}
We have xfs_io precisely so that we don't have to maintain random
test code like this throughout xfstests - do it once, do it right,
use it everywhere.
I totally agree that xfs_io is what people should use, but I disagree on
the use of xfs_io in this particular test, let me explain why.

With 3.16-rc1 at least, it is possible to set fsx_extsize to a non-zero
value, without setting the flags (if you call the ioctl directly). Such
an inode will be (unless I'm mistaken) flagged with a warning by
xfs_repair, which means that it's an invalid inode state.

So in my view, there's a kernel bug, in that it allows a user land
application to put an inode into a "wrong" state. This particular test
is designed to reproduce this kernel bug, so that the kernel fix can be
verified that is indeed a fix.

I can't use xfs_io here, because it will do the "right" thing - set the
EXTSIZE/EXTSZINHERIT flags correctly; but this is a test that the kernel
protects the inode invariants, not that xfs_io does so.

Alternatively, you could say that it's perfectly fine to have a non-zero
fsx_extsize, and that only when the flag is set it should be taken into
account; but I don't think that is what the rest of the code expects
today.

So, I'm fine either way, but I would to fix this so that all the code
agrees what the correct states for an inode are, and that the kernel
prevents user space from violating this assumption via a (documented)
ioctl. Just let me know which are the correct states.

Thanks for the feedback, and for such a quick reply.

iustin
Dave Chinner
2014-08-29 02:52:18 UTC
Permalink
Post by Iustin Pop
Post by Dave Chinner
Post by Iustin Pop
- 307: check that extent size can always be set on a directory
- 308: check that setting a non-zero extent size directly via the ioctl
sets the expected flags (EXTSIZE and EXTSZINHERIT)
OK, will do. There was nothing written in the git repository's README,
hence I chose what I thought best.
Documentation always needs updating, and stuff is in the process of
being moved around.
Post by Iustin Pop
Post by Dave Chinner
Post by Iustin Pop
+# Copyright (c) 2014 Google Inc. All Rights Reserved.
Is that correct? It doesn't match the email address you sent this
always like to check that the copyright assignment is correct in
situations like this...
It is correct indeed (and thanks for double-checking). I prefer to send
my interactions/contributions done not as part of my job using my
but even in that case, the copyright remains with my employer.
Thanks, I'll know in future ;)
Post by Iustin Pop
Post by Dave Chinner
Post by Iustin Pop
+# now create a 'big' (with extents) directory
+mkdir $big
+idx=1
+while xfs_bmap $big | grep -q "no extents"; do
+ touch $big/$idx
+ idx=$((idx+1))
+ if [ "$idx" -gt 1048576 ]; then
+ # still no extents? giving up
+ echo "Can't make a directory to have extents even after 1M files" 1>&2
+ exit
+ fi
+done
urk. largest inode size is 2kb, which means at most it can fit less
than 100 dirents in the inode before spilling to extent form. So
just do a loop that creates 1000 files - there's no need to
overengineer the test code.
Will do. It's fine to still check that the directory does have extents,
I hope?
$XFS_IO_PROG -c 'bmap -vp' $big | _filter_bmap
Post by Iustin Pop
Post by Dave Chinner
Post by Iustin Pop
+int main(int argc, char **argv) {
+ struct fsxattr fa;
+ int fd = open(argv[1], O_RDONLY);
+ if (fd < 0) {
+ perror("open");
+ return 1;
+ }
+ fa.fsx_xflags = 0;
+ fa.fsx_extsize = 1048576 * 8;
+ int r = xfsctl(argv[1], fd, XFS_IOC_FSSETXATTR, &fa);
.... that code is quite broken. Yes, it would work to set the
appropriate extent size flags with the kernel
changes you made, but that's not how this ioctl works.
i.e. it will cause any flag bits that are set on the inode to be
cleared
Good point…
Post by Dave Chinner
and it's likely to fail on old kernels beacuse they have
very different behaviour to what your patch does.
OK, that I didn't know. (Would you mind quickly explaining?)
The extsize hint on the inode is not used by the kernel code unless
the XFS_XFLAG_EXTSIZE is also set. So existing kernels may set the
inode extszhint field, but the code will ignore it because the flags
didn't get set on the inode. See xfs_get_extsz_hint().

With your kernel changes, the above *invalid* code will result in
the flags being set on the inode, and so there's a change of
behaviour from "old kernel, does not trigger extsz behaviour" to
"new kernel, extsz behaviour is invoked".
Post by Iustin Pop
Post by Dave Chinner
We have xfs_io precisely so that we don't have to maintain random
test code like this throughout xfstests - do it once, do it right,
use it everywhere.
I totally agree that xfs_io is what people should use, but I disagree on
the use of xfs_io in this particular test, let me explain why.
With 3.16-rc1 at least, it is possible to set fsx_extsize to a non-zero
value, without setting the flags (if you call the ioctl directly). Such
an inode will be (unless I'm mistaken) flagged with a warning by
xfs_repair, which means that it's an invalid inode state.
Yes, since this commit bd5683f ("xfs_repair: validate inode di_flags
field") in 2011 repair will flag that the hint is non-zero but the
correct flags are not set. Likewise it will warn if the wrong flags
are set. You can get wrong flags set on the inode in many ways, but
historically the kernel and repair utilities haven't cared.
Post by Iustin Pop
So in my view, there's a kernel bug, in that it allows a user land
application to put an inode into a "wrong" state. This particular test
is designed to reproduce this kernel bug, so that the kernel fix can be
verified that is indeed a fix.
Yes this is not what xfstests is for. If we fix an API bug, there's
little value to testing forever that the API is fixed. What we are
trying to cover is that when the parameters are set correctly that
the behaviour is correct.

If you want to do "is the API validating user input correctly"
testing then that's what trinity is for. Dave Jones has long wanted
to get better ioctl coverage for filesystem specific operations, so
I'd suggest that this is the avenue for testing whether an API
behaves correctly and catches all invalid input.

Then we get to fix all the problems that trinity causes, just like
the mm folk have been doing for the past couple of years...

Cheers,

Dave.
--
Dave Chinner
***@fromorbit.com
Continue reading on narkive:
Loading...