xfs_growfs doesn't resize

[k***@sonic.net]

Hello kind XFS folks,

I am having a strange issue with xfs_growfs, and before I attempt to do
something potentially unsafe, I thought I would check in with the list
for advice.

Our fileserver had an ~11TB xfs filesystem hosted under linux lvm. I
recently added more disks to create a new 9TB container, and used the
lvm tools to add the container to the existing volume group. When I
went to xfs_growfs the filesystem, I had the first issue that this user
had, where the metadata was reported, but there was no message about
the new number of blocks:

http://oss.sgi.com/archives/xfs/2008-01/msg00085.html

Fortunately, I have not yet seen the other symptoms that the OP saw: I
can still read from and write to the original filesystem. But the
filesystem size hasn't changed, and I'm not experienced enough to
interpret the xfs_info output properly.

I read through that thread (and others), but none seemed specific to my
issue. Plus, since my filesystem still seems healthy, I'm hoping that
there's a graceful way to resolve the issue and add the new disk space.

Here's some of the information I've seen asked for in the past. I
apologize for it being fairly long.

/proc/partitions:

major minor #blocks name

8 0 244129792 sda
8 1 104391 sda1
8 2 8385930 sda2
8 3 21205800 sda3
8 4 1 sda4
8 5 30876898 sda5
8 6 51761398 sda6
8 7 20555136 sda7
8 8 8233281 sda8
8 9 20603331 sda9
8 16 11718684672 sdb
8 17 11718684638 sdb1
253 1 21484244992 dm-1
8 48 9765570560 sdd
8 49 9765568085 sdd1


sdb1 is the original member of the volume group. sdd1 is the new PV. I
believe dm-1 is the LV where the volume group is hosted (and all the LVM
tools report a 20TB logical volume).


# lvdisplay
--- Logical volume ---
LV Name /dev/saharaVG/saharaLV
VG Name saharaVG
LV UUID DjacPa-p9mk-mBmv-69c2-dmXF-LfxQ-wsRUOD
LV Write Access read/write
LV Status available
# open 1
LV Size 20.01 TB
Current LE 5245177
Segments 2
Allocation inherit
Read ahead sectors 0
Block device 253:1

# uname -a
Linux sahara.xxx 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Yes, it's not a completely current kernel. This box is running CentOS 5
with some yum updates.

# xfs_growfs -V
xfs_growfs version 2.9.4

This xfs_info is from after the xfs_growfs attempt. I regret that I don't have one from before; I was actually thinking of it, but the resize went so smoothly on my test machine (and went fine in the past as well on other platforms) that I didn't give it much thought till it was too late.

# xfs_info /export/
meta-data=/dev/mapper/saharaVG-saharaLV isize=256 agcount=32, agsize=91552192 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=2929670144,
imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0

I saw requests to run xfs_db, but I don't want to mess up the syntax, even if -r should be safe.

Thanks for any help you can provide!

--keith

[k***@sonic.net]

Hello again all,

I apologize for following up my own post, but I found some new information.

Post by k***@sonic.net
http://oss.sgi.com/archives/xfs/2008-01/msg00085.html

I found a newer thread in the archives which might be more relevant to my issue:

http://oss.sgi.com/archives/xfs/2009-09/msg00206.html

But I haven't yet done a umount, and don't really wish to. So, my followup questions are:

==Is there a simple way to figure out what xfs_growfs did, and whether it caused any problems?
==Will I be able to fix these problems, if any, without needing a umount?
==Assuming my filesystem is healthy, will a simple kernel update (and reboot of course!) allow me to resize the filesystem in one step, instead of 2TB increments?

Again, many thanks!

--keith

[Dave Chinner]

Post by k***@sonic.net
Hello again all,
I apologize for following up my own post, but I found some new information.

Post by k***@sonic.net
http://oss.sgi.com/archives/xfs/2008-01/msg00085.html

http://oss.sgi.com/archives/xfs/2009-09/msg00206.html
==Is there a simple way to figure out what xfs_growfs did, and whether it caused any problems?

Apart from looking at what is on disk with xfs_db in the manner that
is done in the first thread you quoted, no.

Post by k***@sonic.net
==Will I be able to fix these problems, if any, without needing a umount?

If you need to modify anything with xfs_db, then you have to unmount
the filesystem first. And realistically, you need to unmount the
filesystem to make sure what xfs-db is reporting is not being
modified by the active filesystem.

So either way, you will have to unmount the filesystem.

Post by k***@sonic.net
==Assuming my filesystem is healthy, will a simple kernel update
(and reboot of course!) allow me to resize the filesystem in one
step, instead of 2TB increments?

I'd upgrade both kernel and userspace.

Cheers,

Dave.

[k***@sonic.net]

Thanks for the response, Dave! I have some additional questions inline.

Post by Dave Chinner
So either way, you will have to unmount the filesystem.

Yikes! I am guessing that may put the filesystem at risk of not being able to re-mount without xfs_db commands, as happened to the other posters I cited. If I want to minimize the amount of downtime if umounting does cause the fs not to be mountable, is there a way for me to look at the xfs_db output after I umount, and calculate any new parameters myself? Or is that considered generally unwise, and xfs_db needs an expert to look at the output? I want to minimize downtime, but I also want to minimize the risk of data loss, so I wouldn't want to derive my own xfs_db commands unless it was very safe. (Even with backups available, it's more work to switch over or restore if I do lose the filesystem; we're a small group so we don't have an automatic failover server.)

Are there any other docs concerning using xfs_db? I saw a post from last year that said that there weren't, but I'm wondering if that's changed since then. There is of course the man page, but that doesn't describe how to interpret what's going on from its output (or what the correct steps to take are if there's a problem).

Post by Dave Chinner

Post by k***@sonic.net
==Assuming my filesystem is healthy, will a simple kernel update
(and reboot of course!) allow me to resize the filesystem in one
step, instead of 2TB increments?

I'd upgrade both kernel and userspace.

Would you recommend upgrading userspace from source? CentOS 5 still calls the version available (from their centosplus repo) 2.9.4, but I haven't investigated what sort of patches they may have applied.


--keith

[Eric Sandeen]

Post by k***@sonic.net
# uname -a
Linux sahara.xxx 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Yes, it's not a completely current kernel. This box is running CentOS 5
with some yum updates.

try

# rpm -qa | grep xfs

If you see anything with "kmod" you're running an exceptionally old xfs codebase.

2.6.18-138 and beyond should have a newer xfs backport built into the kernel
rpm itself, as shipped from Red Hat. But the ancient xfs-kmod (or similar)
provided xfs.ko will take precedence even if you update that kernel.

* Fri Apr 03 2009 Don Zickus <***@redhat.com> [2.6.18-138.el5]
...
- [fs] xfs: update to 2.6.28.6 codebase (Eric Sandeen ) [470845]

If at all possible I'd try an updated kernel, especially if your xfs.ko
is provided by the very, very, very old centos xfs-kmod rpm.

-Eric

[Eric Sandeen]

Post by Eric Sandeen

Post by k***@sonic.net
# uname -a
Linux sahara.xxx 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Yes, it's not a completely current kernel. This box is running CentOS 5
with some yum updates.

try
# rpm -qa | grep xfs
If you see anything with "kmod" you're running an exceptionally old xfs codebase.
2.6.18-138 and beyond should have a newer xfs backport built into the kernel
rpm itself, as shipped from Red Hat. But the ancient xfs-kmod (or similar)
provided xfs.ko will take precedence even if you update that kernel.

... unless you uninstall the xfs-kmod package.

(i'm not sure how to set precedence of found kernel modules, I guess)

-Eric

Post by Eric Sandeen
...
- [fs] xfs: update to 2.6.28.6 codebase (Eric Sandeen ) [470845]
If at all possible I'd try an updated kernel, especially if your xfs.ko
is provided by the very, very, very old centos xfs-kmod rpm.
-Eric

[Eric Sandeen]

Post by k***@sonic.net

Post by Eric Sandeen

Post by k***@sonic.net
# uname -a
Linux sahara.xxx 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Yes, it's not a completely current kernel. This box is running CentOS 5
with some yum updates.

try
# rpm -qa | grep xfs
If you see anything with "kmod" you're running an exceptionally old xfs codebase.

Yes, I do have a kmod-xfs package, so clearly a kernel update is in
order. So my goals are twofold: 1) verify the current filesystem's
state--is it healthy, or does it need xfs_db voodoo? 2) once it's
determined healthy, again attempt to grow the filesystem. Here is
0) get a nearer-term backup, just in case :) The filesystem still seems
perfectly normal, but without knowing what my first xfs_growfs did I
don't know if or how long this state will last.

good idea.

Post by k***@sonic.net
1) umount the fs to run xfs_db
2) attempt a remount--is this safe, or is there risk of damaging the
filesystem?

I'm not sure.

You probably hit this bug:
http://oss.sgi.com/archives/xfs/2007-01/msg00053.html

See also:
http://oss.sgi.com/archives/xfs/2009-07/msg00087.html

I can't remember how much damage the original bug did ...

Post by k***@sonic.net
3) If a remount succeeds, then update the kernel and xfsprogs. If
a remount doesn't work, then revert to the near-term backup I took
in 0) and attempt to fix the issue (with the help of the list, I hope).

One thing you might be able to do, though I don't remember for sure
if this works, is to freeze the fs and create an xfs_metadump
image of it. You can then point xfs_repair at that image, and see
what it finds. But I'm not sure if metadump will work on a frozen
fs... hm no. Only if it's mounted ro.

Otherwise -maybe- xfs_repair -n -d might work after a mount -o remount,ro.

(-n -d means operate in no-modify mode on an ro-mounted fs)


So you'd need to mount readonly before you could either do xfs_repair -nd
or xfs_metadump followed by repair of that image. Either one would give
you an idea of the health of the fs.

Post by k***@sonic.net
4) In either case, post my xfs_db output to the list and get your
opinions on the health of the fs.

repair probably will tell you more as an initial step.

Post by k***@sonic.net
5) If the fs seems correct, attempt xfs_growfs again.
Do all these steps seem reasonable? I am most concerned about step 2--
I really do want to be able to remount as quickly as possible, but I
do not know how to tell whether it's okay from xfs_db's output. So if a
remount attempt is reasonably nondestructive (i.e., it won't make worse
an already unhealthy XFS fs) then I can try it and hope for the best.
(From the other threads I've seen it seems like it's not a good idea to
run xfs_repair.)

you can run it with -n to do no-modify. If it's clean, you're good;
if it's a mess, you won't hurt anything, other than making you sad. :)

Post by k***@sonic.net
Would it make more sense to update the kernel and xfsprogs before
attempting a remount? If a remount fails under the original kernel,

is it still mounted I guess?

A newer up to date kernel certainly won't make anything -worse-

You should uninstall that kmod rpm though so it doesn't get priority
over the xfs.ko in the new kernel. If you need to revert to the old
kernel, you could always reinstall it.

Post by k***@sonic.net
what do people think the odds are that a new kernel would be able to
mount the original fs, or is that really unwise?

I don't think a newer kernel would do any further harm.

Post by k***@sonic.net
Again, many thanks for all your help.

You're welcome but here's the obligatory plug in return - running RHEL5
proper would have gotten you up to date, fully supported xfs, and you wouldn't
have run into this mess. Just sayin' ... ;)

-Eric

Post by k***@sonic.net
--keith

[k***@sonic.net]

Post by Eric Sandeen

Post by k***@sonic.net
# uname -a
Linux sahara.xxx 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Yes, it's not a completely current kernel. This box is running CentOS 5
with some yum updates.

try
# rpm -qa | grep xfs
If you see anything with "kmod" you're running an exceptionally old xfs codebase.

Yes, I do have a kmod-xfs package, so clearly a kernel update is in
order. So my goals are twofold: 1) verify the current filesystem's
state--is it healthy, or does it need xfs_db voodoo? 2) once it's
determined healthy, again attempt to grow the filesystem. Here is
my current plan for reaching these goals:

0) get a nearer-term backup, just in case :) The filesystem still seems
perfectly normal, but without knowing what my first xfs_growfs did I
don't know if or how long this state will last.

1) umount the fs to run xfs_db

2) attempt a remount--is this safe, or is there risk of damaging the filesystem?

3) If a remount succeeds, then update the kernel and xfsprogs. If a remount
doesn't work, then revert to the near-term backup I took in 0) and attempt
to fix the issue (with the help of the list, I hope).

4) In either case, post my xfs_db output to the list and get your
opinions on the health of the fs.

5) If the fs seems correct, attempt xfs_growfs again.

Do all these steps seem reasonable? I am most concerned about step 2--
I really do want to be able to remount as quickly as possible, but I
do not know how to tell whether it's okay from xfs_db's output. So if a
remount attempt is reasonably nondestructive (i.e., it won't make worse
an already unhealthy XFS fs) then I can try it and hope for the best.
(From the other threads I've seen it seems like it's not a good idea to
run xfs_repair.)

Would it make more sense to update the kernel and xfsprogs before
attempting a remount? If a remount fails under the original kernel,
what do people think the odds are that a new kernel would be able to
mount the original fs, or is that really unwise?

Again, many thanks for all your help.

--keith

[Eric Sandeen]

[some rearranging]

Post by Eric Sandeen
You're welcome but here's the obligatory plug in return - running
RHEL5 proper would have gotten you up to date, fully supported xfs,
and you wouldn't have run into this mess. Just sayin' ... ;)

Yep, that's definitely a lesson learned. Though I don't think I can
blame CentOS either--from what I can tell the bug has been available
from yum for some time now. So it's pretty much entirely my own
fault. :(

well it's unfortunate that that kmod persists. I'll admit to providing
it, years and years ago... Centos should find a way to deprecate it...

I also am sorry for not preserving threading--for some reason, the
SGI mailserver rejected mail from my normal host (which is odd, as
it's not in any blacklists I know of), so I am using an unfamiliar
mail client.

sgi email ... sucks ;)

Post by Eric Sandeen
http://oss.sgi.com/archives/xfs/2007-01/msg00053.html [1]
See also: http://oss.sgi.com/archives/xfs/2009-07/msg00087.html
[2]
I can't remember how much damage the original bug did ...

If any? I'm a bit amazed that, if there was damage, that the
filesystem is still usable. Perhaps if I were to fill it it would
show signs of inconsistency? Or remounting would read the
now-incorrect values from the superblock 0?

Post by Eric Sandeen
is it still mounted I guess?

Yes, it's still mounted, and as far as I can tell perfectly fine.
But I won't really know till I can throw xfs_repair -n and/or xfs_db
and/or remount it; I'm choosing to get as much data off as I can
before I try these things, just in case.
How safe is running xfs_db with -r on my mounted filesystem? I

it's safe. At worst it might read inconsistent data, but it's
perfectly safe.

understand that results might not be consistent, but on the off
chance that they are I am hoping that it might be at least a little
helpful.
I was re-reading some of the threads I posted in my original
http://oss.sgi.com/archives/xfs/2009-09/msg00210.html
http://oss.sgi.com/archives/xfs/2009-09/msg00211.html
If I am reading those, plus the xfs_db man page, correctly, it seems
like what Russell suggested was to look at superblock 1 (or some
other one?) and use those values to correct superblock 0. At what

don't worry about correcting anything until you know there is a problem :)

points (if any) are the other superblocks updated? I was testing on
another machine, on a filesystem that I had successfully grown using
xfs_growfs, and of the two values Russell suggested the OP to change,
dblocks is different between sb 0 and sb 1, but agcount is not.
Could that just be that I did not grow the filesystem too much, so
that agcount didn't need to change? That seems a bit
counterintuitive, but (as should be obvious) I don't know XFS all

if you grew it 9T, you would have almost certainly gotten more AGs.
If you did a smaller test then you might see that. To be honest
I don't remember when the backup superblocks get updated.

that well. I am hoping to know because, in re-reading those
messages, I got a better idea of what those particular xfs_db
commands do, so that if I did run into problems remounting, I might
be able to determine the appropriate new values myself and reduce my
downtime. But I want to understand more what I'm doing before I try
that!

I think finding a way to do a dry-run xfs_repair would be the best
place to start ...

Get a recent xfsprogs too, if you haven't already, it scales better
than the really old versions.

-Eric

--keith

[k***@sonic.net]

On Sun 03/07/11 3:14 PM , Eric Sandeen <***@sandeen.net> wrote:

[some rearranging]

Post by Eric Sandeen
You're welcome but here's the obligatory plug in return - running RHEL5
proper would have gotten you up to date, fully supported xfs, and you
wouldn't have run into this mess. Just sayin' ... ;)

Yep, that's definitely a lesson learned. Though I don't think I can blame CentOS either--from what I can tell the bug has been available from yum for some time now. So it's pretty much entirely my own fault. :(

I also am sorry for not preserving threading--for some reason, the SGI mailserver rejected mail from my normal host (which is odd, as it's not in any blacklists I know of), so I am using an unfamiliar mail client.

Post by Eric Sandeen
http://oss.sgi.com/archives/xfs/2007-01/msg00053.html [1]
http://oss.sgi.com/archives/xfs/2009-07/msg00087.html [2]
I can't remember how much damage the original bug did ...

If any? I'm a bit amazed that, if there was damage, that the filesystem is still usable. Perhaps if I were to fill it it would show signs of inconsistency? Or remounting would read the now-incorrect values from the superblock 0?

Post by Eric Sandeen
is it still mounted I guess?

Yes, it's still mounted, and as far as I can tell perfectly fine. But I won't really know till I can throw xfs_repair -n and/or xfs_db and/or remount it; I'm choosing to get as much data off as I can before I try these things, just in case.

How safe is running xfs_db with -r on my mounted filesystem? I understand that results might not be consistent, but on the off chance that they are I am hoping that it might be at least a little helpful.

I was re-reading some of the threads I posted in my original messages, in particular these posts:

http://oss.sgi.com/archives/xfs/2009-09/msg00210.html
http://oss.sgi.com/archives/xfs/2009-09/msg00211.html

If I am reading those, plus the xfs_db man page, correctly, it seems like what Russell suggested was to look at superblock 1 (or some other one?) and use those values to correct superblock 0. At what points (if any) are the other superblocks updated? I was testing on another machine, on a filesystem that I had successfully grown using xfs_growfs, and of the two values Russell suggested the OP to change, dblocks is different between sb 0 and sb 1, but agcount is not. Could that just be that I did not grow the filesystem too much, so that agcount didn't need to change? That seems a bit counterintuitive, but (as should be obvious) I don't know XFS all that well. I am hoping to know because, in re-reading those messages, I got a better idea of what those particular xfs_db commands do, so
that if I did run into problems remounting, I might be able to determine the appropriate new values myself and reduce my downtime. But I want to understand more what I'm doing before I try!
that!

--keith

[k***@sonic.net]

Hello again XFS folks,

I have finally made the time to revisit this, after copying most of my data
elsewhere.

How safe is running xfs_db with -r on my mounted filesystem? I

it's safe. At worst it might read inconsistent data, but it's
perfectly safe.

So, here is my xfs_db output. This is still on a mounted filesystem.

# xfs_db -r -c 'sb 0' -c 'print' /dev/mapper/saharaVG-saharaLV
magicnum = 0x58465342
blocksize = 4096
dblocks = 5371061248
rblocks = 0
rextents = 0
uuid = 1bffcb88-0d9d-4228-93af-83ec9e208e88
logstart = 2147483652
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 91552192
agcount = 59
rbmblocks = 0
logblocks = 32768
versionnum = 0x30e4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 27
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 19556544
ifree = 1036
fdblocks = 2634477046
frextents = 0
uquotino = 131
gquotino = 132
qflags = 0x7
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0


# xfs_db -r -c 'sb 1' -c 'print' /dev/mapper/saharaVG-saharaLV
magicnum = 0x58465342
blocksize = 4096
dblocks = 2929670144
rblocks = 0
rextents = 0
uuid = 1bffcb88-0d9d-4228-93af-83ec9e208e88
logstart = 2147483652
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 91552192
agcount = 32
rbmblocks = 0
logblocks = 32768
versionnum = 0x30e4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 27
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 19528640
ifree = 15932
fdblocks = 170285408
frextents = 0
uquotino = 131
gquotino = 132
qflags = 0x7
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0


I can immediately see with a diff that dblocks and agcount are
different. Some other variables are also different, namely icount,
ifree, and fdblocks, which I am unclear how to interpret. But judging
from the other threads I quoted, it seems that dblocks and agcount
are using values for a 20TB filesystem, and that therefore on a umount
the filesystem will become (at least temporarily) unmountable.

I've seen two different routes for trying to correct this issue--either use
xfs_db to manipulate the values directly, or using xfs_repair on a frozen
ro-mounted filesystem with a dump from xfs_metadata. My worry about
the latter is twofold--will I even be able to do a remount? And will I
have space for a dump from xfs_metadata of an 11TB filesystem? I have
also seen advice in some of the other threads that xfs_repair can actually
make the damage worse (though presumably xfs_repair -n should be safe).

If xfs_db is a better way to go, and if the values xfs_db returns on a
umount don't change, would I simply do this?

# xfs_db -x /dev/mapper/saharaVG-saharaLV
sb 0 w dblocks = 2929670144 w agcount = 32

and then do an xfs_repair -n?

A route I have used many ages ago, on ext2 filesystems, was to specify
an alternate superblock when running e2fsck. Can xfs_repair do this?

Get a recent xfsprogs too, if you haven't already, it scales better
than the really old versions.

I think I may have asked this in another post, but would you suggest
compiling 3.0 from source? The version that CentOS distributes is marked
as 2.9.4, but I don't know what patches they've applied (if any). Would 3.0
be more likely to help recover the fs?

Thanks all for your patience!

--keith

[Keith Keller]

Hi all,

First, I hope that this message fixes my mail client breaking threading.

I am sorry for following up my own post (again), but I realized this

Post by k***@sonic.net
So, here is my xfs_db output. This is still on a mounted filesystem.

How safe/risky is it to leave this filesystem mounted and in use?
I'm not too concerned about new data, since it won't be a huge amount,
but I am wondering if data that's already been written may be at risk.
Or, it it a reasonable guess that the kernel is still working completely
with the old filesystem geometry, and so won't write anything beyond the
old limits while it's still mounted? df certainly seems to use the old
fs size, not the new one.

Thanks again,


--keith

[Eric Sandeen]

Post by Keith Keller
Hi all,
First, I hope that this message fixes my mail client breaking threading.
I am sorry for following up my own post (again), but I realized this

Post by k***@sonic.net
So, here is my xfs_db output. This is still on a mounted filesystem.

How safe/risky is it to leave this filesystem mounted and in use?
I'm not too concerned about new data, since it won't be a huge amount,
but I am wondering if data that's already been written may be at risk.
Or, it it a reasonable guess that the kernel is still working completely
with the old filesystem geometry, and so won't write anything beyond the
old limits while it's still mounted? df certainly seems to use the old
fs size, not the new one.

I don't remember all the implications of this very old bug...

It seems like you need an "exit strategy" - you probably cannot leave
your fs mounted -forever- ...

If it were me, if possible, I'd make backups of the fs as it's mounted
now, then umount it and make an xfs_metadump of it, restore that metadump
to a new sparse file, and point xfs_repair at that metadata image file,
to see what repair might do with it.

If repair eats it alive, then we can look into more xfs_db type surgery
to fix things up more nicely...

-Eric

Post by Keith Keller
Thanks again,
--keith

[Eric Sandeen]

...

Post by Eric Sandeen
If it were me, if possible, I'd make backups of the fs as it's mounted
now, then umount it and make an xfs_metadump of it, restore that metadump
to a new sparse file, and point xfs_repair at that metadata image file,
to see what repair might do with it.
If repair eats it alive, then we can look into more xfs_db type surgery
to fix things up more nicely...

This sounds like a reasonable plan. It looks like xfs_metadump needs a
umounted or readonly filesystem in order to work properly; is there any
way to estimate how long such a dump would take, and how large it would
be from an almost-full 11TB fs with however many inodes it has (~19 million
IIRC)? I want to plan downtime and usable disk space accordingly.

well, I'm looking at an image of a 4T fs right now, with 208k inodes,
and the image itself took up 800M (a 4T sparse file when restored,
of course, but only using 800M)

Would xfs_metadump create the same dump from a filesystem remounted ro
as from a filesystem not mounted? I think you suggested this idea in

yes, looks like it works, with recent tools anyway.

an earlier post. In a very optimistic scenario, I could imagine
remounting the original filesystem ro, taking the metadump, then being
able to remount rw so that I could put it back into service while I
work with the metadump. Then, once I knew more about the metadump, I

I think that should work.

could do an actual umount and fix the filesystem using the information
gathered from the metadump testing. If they will produce the same
metadump, then it could be a win-win if it's able to successfully
remount rw afterwards; and if it can't, it wasn't any additional effort
or risk to try.

agreed.

Will xfsprogs 3.1.5 work with the older kernel, and will it make a
better dump than 2.9.4? I have built xfsprogs from source, but if it

2.9.4 won't have xfs_metadump ... and no problems with newer tools on
older kernels. It's just reading the block device, in any case.
No unique kernel interaction.

might have problems working with the kmod-xfs kernel module I can use
the 2.9.4 tools instead. (Again, in keeping with the hope-for-the-best
scenario above, if avoiding a reboot won't be too harmful it'd be
convenient.)

I think you can.

I think you also mentioned that xfs_metadump can not dump frozen
filesystems, but the man page for 3.1.5 says it can. FWIW xfs_metadump
refused to work on a frozen filesystem on my test machine, which is
version 2.9.4 (though from an older CentOS base). (xfs_freeze does look
like a nice tool though!)

it should(?) but:

# xfs_metadump /dev/loop0 blah
xfs_metadump: /dev/loop0 contains a mounted and writable filesystem

fatal error -- couldn't initialize XFS library

# xfs_freeze -f mnt/
# xfs_metadump /dev/loop0 blah
xfs_metadump: /dev/loop0 contains a mounted and writable filesystem

fatal error -- couldn't initialize XFS library

# xfs_freeze -u mnt
# mount -o remount,ro mnt
# xfs_metadump /dev/loop0 blah

<proceeds w/o problems>

I think we should make the tools work with freeze, but remount,ro works fine too.

-Eric

--keith

[Keith Keller]

Hello again list!

Back from my vacation, with no problems on the filesystem I
potentially messed up by running xfs_growfs with an old kernel
a few weeks ago. I was able to take a metadump of the filesystem
by remounting ro, running xfs_metadump, and remounting rw, which did
not seem to cause any obvious problems. Since then I have been
throwing the various xfs tools at it to see what I could find out.

The last thing I tried gave an interesting result: I simply
did an mdrestore and attempted a mount over loopback, on a
different machine that is running the latest kernel available
from the CentOS repository (2.6.18-238.9.1.el5). To my great
surprise, it mounted cleanly, and reported the correct filesystem
size! A very cursory glance at the filesystem didn't uncover
any glaring errors (though I forgot to use -o when I ran
xfs_metadump, so filenames are mangled).

The reason I tried this at all was that I got some unexpected
results from xfs_repair in various permutations. First, just
running it against the restored fs image, I got this error (using
a locally-compiled xfsprogs-3.1.5, not the xfsprogs CentOS
package--all references to xfsprogs programs will be this version
unless stated otherwise):

Phase 1 - find and verify superblock...
would write modified primary superblock
Primary superblock would have been modified.
Cannot proceed further in no_modify mode.
Exiting now.

That wasn't a huge surprise, so I reran xfs_repair without -n, and
got many errors like so:

bad directory block magic # 0 in block 0 for directory inode 16359537
corrupt block 0 in directory inode 16359537
will junk block

When I examined the files added to lost+found, there were a large number
of them, but they all were owned by the same user and generated within
a few days of each other. (Are owner and group UIDs preserved by a
metadump? It would seem so.) I thought this was curious, and wasn't
sure exactly what was going on, so I ran the xfs_db alterations that
I asked about in a previous post (basically resetting dblocks and agcount
to the values from sb 1) and ran xfs_repair on the modified image. This
time, running with -n succeeded fine, but reported that it would fix
almost the same set of errors that the first xfs_repair did report that
it tried to fix. So, I am wondering if those files are truly damaged,
and something else caused the filesystem inconsistency, not xfs_growfs.
(The dates on the files in lost+found are about a week after we had a
power outage, but months before my failed xfs_growfs attempt.)

At this point, I really don't know what to try next. I still have the
metadump, so I can do pretty much anything I want to it to try to narrow
down the best course of action on the actual filesystem. What information
would be most helpful for me to report? If it would help to see the
actual output from xfs_repair, it's almost certainly too big to post to
the list, but I can make them available over http.

Just for posterity's sake, xfs_metadump on my almost-full 11TB filesystem
took about 15 minutes, and generated a ~6.7GB dump file. Restoring this
dump to a sparse file takes about 4 minutes. xfs_repair takes about 30-45
minutes on these images (I haven't been tracking the times that
carefully). I'm sure running these tools on an actual 11TB filesystem
would take a whole lot longer, so having the metadump is really a great
tool for this kind of work.

Thanks for reading!

--keith

[Keith Keller]

Post by Eric Sandeen
It seems like you need an "exit strategy" - you probably cannot leave
your fs mounted -forever- ...

Yes, of course. I didn't mean to imply that I'd leave it like this
indefinitely. :)

I will be on vacation next week, and it's not really reschedulable. So
my plan was to ride the filesystem through next week, hope for the best,
then fix it when I return, rather than attempt to start a fix now and
risk ending up with a broken filesystem. Ideally I would preemptively
switch to a warm backup before I leave, but that won't be ready in time.
(I currently do have all the important data backed up, but it is to
various different spaces where I had free disk space. The warm backup
should be ready early next week if the filesystem does go belly-up
before I return.)

Post by Eric Sandeen
If it were me, if possible, I'd make backups of the fs as it's mounted
now, then umount it and make an xfs_metadump of it, restore that metadump
to a new sparse file, and point xfs_repair at that metadata image file,
to see what repair might do with it.
If repair eats it alive, then we can look into more xfs_db type surgery
to fix things up more nicely...

This sounds like a reasonable plan. It looks like xfs_metadump needs a
umounted or readonly filesystem in order to work properly; is there any
way to estimate how long such a dump would take, and how large it would
be from an almost-full 11TB fs with however many inodes it has (~19 million
IIRC)? I want to plan downtime and usable disk space accordingly.

Would xfs_metadump create the same dump from a filesystem remounted ro
as from a filesystem not mounted? I think you suggested this idea in
an earlier post. In a very optimistic scenario, I could imagine
remounting the original filesystem ro, taking the metadump, then being
able to remount rw so that I could put it back into service while I
work with the metadump. Then, once I knew more about the metadump, I
could do an actual umount and fix the filesystem using the information
gathered from the metadump testing. If they will produce the same
metadump, then it could be a win-win if it's able to successfully
remount rw afterwards; and if it can't, it wasn't any additional effort
or risk to try.

Will xfsprogs 3.1.5 work with the older kernel, and will it make a
better dump than 2.9.4? I have built xfsprogs from source, but if it
might have problems working with the kmod-xfs kernel module I can use
the 2.9.4 tools instead. (Again, in keeping with the hope-for-the-best
scenario above, if avoiding a reboot won't be too harmful it'd be
convenient.)

I think you also mentioned that xfs_metadump can not dump frozen
filesystems, but the man page for 3.1.5 says it can. FWIW xfs_metadump
refused to work on a frozen filesystem on my test machine, which is
version 2.9.4 (though from an older CentOS base). (xfs_freeze does look
like a nice tool though!)

--keith