File changed as we read it GlusterFS issue

File changed as we read it GlusterFS issue

File changed as we read it GlusterFS issue

Recently i had trouble running Backups of my Gitlab instance onto GlusterFS mounted volumes. The tar cronjob always exited with error message: File changed as we read it GlusterFS issue. So i was wondering what is going on and found a 3 year old bug report which describes the problem. They made a workaround for that behavior which i will describe here.

First of all the problem is as follows: Tar grabs the metadata stats from files which it is going to backup and after the backup has happen it verifies those metadata again and fails. Due to design that Glusterfs operates on multiple bricks, it occurs that the connection goes to another brick than previously were the atime/mtime can be different which produces this behavior.

Prerequisites:

• A working GlusterFS 3.7+ Setup

• Any texteditor (vi, joe, nano) of your choice

If you are using the native glusterfs fuse client, the only way to prevent this is by enabling following patch:

gluster volume set Put-your-netshare-here cluster.consistent-metadata on

In fact the workaround that they implemented is not the best solutions, since it decreases performance of the fuse mount even more.
A better approach would be to implement an option in fuse mount to set: noatime, nodiratime which isnt supported yet.

If you are using the gNfs implementation of GlusterFS you should not have such problems but noatime,nodiratime cant harm:

Put following in your /etc/fstab and change your Mount and ServerIP settings so that they fit to your setup.

10.0.0.1:/netshare/www /netshare/www nfs noauto,rw,acl,async,vers=3,noatime,nodiratime,rs
ize=1048576,wsize=1048576,hard,intr,retrans=2,tcp,nolock,actimeo=1,local_lock=none 0 0

I also added some extra tunings which i usually use on all my production setups to get highest performance and throughput out of my nfs glusterfs mounts.

If you need any assistance feel free and drop me a line!