This is one of the weirdest problems I’ve had for a while. On one of the servers I look after my home directory was a zfs filesystem set to /home/user. If I ever did an rm -rf on a directory structure underneath my home dir, it would remove all the files but leave all directories in place. The only way to get rid of the directories was to recursively remove each one. Painful.
Today I discovered a fix – although I don’t know the reason why.
I had a fairly restrictive umask set, system wide, in /etc/profile – 027. When I did ‘zfs set mountpoint=blah’ with that umask I wouldn’t be able to rm -rf successfully. However, if I moved the mountpoint, set umask to 022, then put it back where it should be, rm -rf works fine!
The underlying directory was left when I temporarily shifted the home directory, and it was owned root:root and 750, as the umask would suggest, but why that would then effect the ZFS mount on top of it (which was correctly owned by my own user ID) I do not know.
Anyone know why this would have been?
Out of interest, what zfs version are you using?
Hi Tom. It’s 4 on this server with the problem.
Cheers
Check your sub-directory ACLs with ‘ls -Vd dir_name’
Maybe this is an issue stemming from the aclinherit property of the ZFS filesystem ?
Works for me by setting setting aclinherit to ‘discard’.
Here’s a test directory Tom:
test@test ~ $ ls -vd test/
drwxr-xr-x 3 test other 3 Apr 27 16:06 test/
0:owner@::deny
1:owner@:list_directory/read_data/add_file/write_data/add_subdirectory
/append_data/write_xattr/execute/write_attributes/write_acl
/write_owner:allow
2:group@:add_file/write_data/add_subdirectory/append_data:deny
3:group@:list_directory/read_data/execute:allow
4:everyone@:add_file/write_data/add_subdirectory/append_data/write_xattr
/write_attributes/write_acl/write_owner:deny
5:everyone@:list_directory/read_data/read_xattr/execute/read_attributes
/read_acl/synchronize:allow
What did you do to set aclinherit to discard?
Cheers
OK, I see -
root@test /home/test # zfs get aclinherit tank/test
NAME PROPERTY VALUE SOURCE
tank/test aclinherit discard local
made no difference for me though :( It was still a case of sorting the umask first.
Try this – set umask 027, create a new FS for a home dir, become the user, create a series of embedded directories, and then rm -rf the top one. Does they all delete, or do they stay right put?
Helpfully, if I use rm without the ‘f’ flag (which , though handy, suppresses errors) I see :
“rm: cannot determine if this is an ancestor of the current working directory”
Why does a recursive rm need to examine the parent directory? The only clue we get is this opaque comment in the latest rm.c sourcecode :
/*
* If we can’t stat “..”, it’s either not there or we can’t search
* the current directory; in that case we can’t return back through
* “..”, so we need to keep the parent open.
* Check that we came from “..”, if not then this directory entry is an
* additional link and there is risk of a filesystem cycle and we also
* can’t go back up through “..” and we keep the directory open.
*/
It’s tempting to reopen bugID# 4677347.
Yes the aclinherit was a red herring, sorry.
If you become the user and try to ‘ls -l’ the zfs mountpoint
*oops, I hit enter too soon*
If you become the user and do ‘ls -la’ of the ZFS mountpoint you also get a “Permission denied” error.
This looks to be spot on :
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6389212
Sounds like it. However, that’s very old (build 36) and I would’ve thought a patch I’ve applied by now would’ve sorted the issue out. Maybe not though.
I hit the same issue with umask se to both 027 and 022. The following allows me to delete the files and directories.
rm -rf
find -depth -exec rmdir ‘[]‘ \;
rm -rf returns code 2 and rmdir returns code 0.
Deleting the directory as root works fine.
Thanks Matt – that’s pretty much the way I was dealing with it before I found the root cause though. Nice to have a workaround, nicer to know what’s causing the issue I always think.
Here is your answer!!!
If you are mounting a zpool, say for instance TANK… and you have its mountpoint set to /app
you have a bunch of zfs partitions created on top of it and you are having this user recursive removal issue (i.e. user cant use rm -rf /app/bob) do this.
zpool export -f TANK
chmod 755 /app
zpool import TANK
This Should Work!
We had similar situation. Root’s umask was set to 027 so that when ZFS auto-created mount dirs, they were set 750. Exported, fixed the perms, re-imported and now we’re good. Thanks for the tip!
…and set root’s umask to 022
Ace, glad my post was of use Paul. It’s a slightly annoying bug actually, one I wish Sun [Oracle] would address!