Re: autofs mounts expiring unexpectedly and unable to remount after disappearing
Ian Kent <[email protected]>
| Newsgroups | org.kernel.vger.autofs |
|---|---|
| Message-ID | <[email protected]> |
On 12/6/24 02:40, Nathaniel Middleton wrote:
> Summary:
> We define a /mount/path/pathcontd/mountpoint tree in autofs, have historically --ghosted the paths, but have recently attempted to set the offset mounts as browse-able.
> Our configuration works great... at first. Once the timeout hits (I set this to 10s for debugging... but normally have it set to 300s), the autofs (type) mount points start expiring. Once this expiration starts, the offset mounts disappear (often leaving one mount point remaining that is broken). The disappeared mount points are unable to be re-mounted. This has hit us at semi-regular intervals on Centos 7 , but Rocky 8 servers are highly repro-able. This could be due to the lack of activity on the new systems though.
Right, the description sounds like there's been a partial expire of the
multi-mount (or possibly only
a directory has not been removed).
There are a couple of things to be aware of, first these multi-mounts
will create the sub-directory tree
below them and remove it when thet expire, that's normal behviour. Only
the root of the map entry is affected
by the browse setting.
>
> Config:
> distro Name: Rocky Linux 8.8 (Green Obsidian)
> autofs-5.1.4-102.el8_8.2
> NFS4.{0..2}(though... the nfs mounts are not being invoked when this issue happens.)
> Autofs configs:
> Auto.master:
> /allen/programs/celltypes /etc/auto.programs_celltypes tcp hard intr timeo=600 retrans=2 async --ghost
>
> auto.programs_celltypes(snipping out similar lines to conserve space):
> production -fstype=nfs4 /incoming aidc-isi1-prd:/ifs/incoming/aibs/celltypes \
> /0378 aidc-isi1-prd:/ifs/programs/celltypes/production/0378 \
> /brainstim aidc-isi1-prd:/ifs/programs/celltypes/production/brainstim \
> /conn aidc-isi1-prd:/ifs/programs/celltypes/production/conn
> workgroups /0285 aidc-isi1-prd:/ifs/programs/celltypes/workgroups/0285 \
> /0378 aidc-isi1-prd:/ifs/programs/celltypes/workgroups/0378
>
>
> We have a couple similar mounts in a similar mount point to "/allen/programs/<different name>/<mount>" if it makes a difference there.
>
> What this looks like:
>
> Temporarily working:
> ls /allen/programs/celltypes/production
> incoming 0378 brainstim conn
Sure that;s what the directory should look like upon access.
>
> Broken:
> ls /allen/programs/celltypes/production
> 0378
And this is what it will look like on a partial expire.
That directory remaining will stop the automount being mounted again and
make it pretty much
useless from this point on.
What we need to do is work out what's causing the partial expire and fix it.
That can be quite difficult.
There usually is some evidence of an error occurring in full debug logs
and I need that.
I can try to duplicate it with your example mount but I suspect it won't
happen when I try it.
I'll have a look at the RHEL revision 102 and see if I can spot anything
too.
>
> Statting of this path results in a wedged terminal, the autofs mount still is showing however:
> /etc/auto.programs_celltypes on /allen/programs/celltypes/workgroups/0378 type autofs (rw,relatime,fd=105,pgrp=4816,timeout=0,minproto=5,maxproto=5,offset,pipe_ino=83070
> I claim that the autofs mounts should not be expiring at all... and that I want mounted NFS sessions to umount after timeout. Stopping autofs results in a loop of trying to umount the remaining autofs shares until it hits a timeout and gets killed.
> The remaining autofs mounts return busy when umount -a -t autofs (or the session locks up). lsof and fuser show 0 processes using the mount point.
> 4 D root 7929 7805 0 80 0 - 6329 autofs 09:37 pts/2 00:00:00 umount -a -t autofs
I think you'll find that you need to be a bit more careful on umounting
a sub-tree of mounts because of the order umount uses,
If the maps haven't been changed you should be able to umount the tree
in the correct order with "automount --force" provided
automount has been terminated already.
>
> If left alone, the state seems to get continually worse as more autofs mounts expire. Example being on the /allen/programs/celltypes/production path (not a NFS mount point)
> ls ing:
> /allen/programs/celltypes/production:
> ^C
Once a partial expire happens it's more or less fatal, all we can do is
try and identify the case
causing it and fix it, if we can.
>
> Logs:
> Jun 5 10:11:49 n238 automount[19033]: expired /allen/programs/celltypes/production
> Jun 5 10:11:49 n238 automount[19033]: expiring path /allen/programs/celltypes/production
> Jun 5 10:11:49 n238 automount[19033]: umount_multi: path /allen/programs/celltypes/production incl 1
> Jun 5 10:11:49 n238 automount[19033]: expired /allen/programs/celltypes/production
> Jun 5 10:11:49 n238 automount[19033]: expiring path /allen/programs/celltypes/production
> Jun 5 10:11:49 n238 automount[19033]: umount_multi: path /allen/programs/celltypes/production incl 1
> Jun 5 10:11:49 n238 automount[19033]: expired /allen/programs/celltypes/production
> Jun 5 10:21:57 n238 automount[25053]: expire_proc_indirect: expire /allen/programs/celltypes/production
> Jun 5 10:24:27 n238 automount[25053]: expire_proc_indirect: expire /allen/programs/celltypes/production
> Jun 5 10:26:57 n238 automount[25053]: expiring path /allen/programs/celltypes/production
> Jun 5 10:26:57 n238 automount[25053]: umount_multi: path /allen/programs/celltypes/production incl 1
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/0378
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/0378
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/brainstim
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/brainstim
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/conn
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/conn
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/ctyconn
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/ctyconn
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/general
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/general
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/histformfish
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/histformfish
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/humancelltypes
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/humancelltypes
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/incoming
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/incoming
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mfish
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mfish
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mfishhisthumanatlas
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mfishhisthumanatlas
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mousecelltypes
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mousecelltypes
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mousegenetictools
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mousegenetictools
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/nhpcelltypes
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/nhpcelltypes
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/synapticphys
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/synapticphys
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/whbi
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/whbi
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/wijem
> Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/wijem
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/brainstim
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/0378
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/humancelltypes
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/histformfish
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/incoming
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mfishhisthumanatlas
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mousegenetictools
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mousecelltypes
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/synapticphys
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/nhpcelltypes
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/wijem
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/whbi
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mfish
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/general
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/ctyconn
> Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/conn
> Jun 5 10:26:57 n238 automount[25053]: expired /allen/programs/celltypes/production
> Jun 5 10:26:58 n238 automount[25053]: expire_proc_indirect: expire /allen/programs/celltypes/production
Don't see any evidence of a problem having occurred here.
Normally I ask for a full debug log (logging = debug in
/etc/autofs.conf) from startup of automount until
the problem occurs. You can send it to me privately if you don't want to
post it.
Ian