`rsync` and FAT File-Systems

I’m using rsync to sync files from my computer to a FAT formatted SD card. Using the --update flag to rsync makes it “skip files that are newer on the receiver”. It seems that it should work as follows: After the first sync, any subsequent syncs will only transfer those files which changed in the meantime. However, I noticed that transfer times are usually longer than expected, which led me to think that things are not working as they should.

As --update relies only on modification date, obviously something is wrong with it. After ruling out several other possibilities, I’ve guessed that the modification date of some files get mangled. A quick check about FAT revealed that FAT can only save modification times in 2 second granularity. I’ve also used rsync’s --archive flag, which among other things attempts to preserve modifications times of the files it copies. But what about FAT’s lower granularity for modification time? That was apparently the culprit. Some files when copied got a modification time which was up to 1 second before the original modification time! Hence, every time I’ve synced, from rsync’s perspective, the target was older than the source, and hence needs to be overwritten.

This can be demonstrated by the following session:

$ dd if=/dev/urandom of=test-even bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 6.20716 s, 16.9 MB/s
$ cp test-even test-odd

$ touch -t 201701180000.00 test-even 
$ touch -t 201701180000.01 test-odd

$ stat -c %Y test-*
1484690400
1484690401

We’ve created two files test-even and test-odd with random content. We’ve set the modification time of test-even to be on an even second, and for test-odd to be on an odd second (one second apart). Simply syncing the files to a FAT partitioned device works as expected:

$ rsync -av --update test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
test-even
test-odd

sent 209,766,550 bytes  received 54 bytes  7,627,876.51 bytes/sec
total size is 209,715,200  speedup is 1.00

Resyncing immediately also works:

$ rsync -avv --update test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test-even is uptodate
test-odd is uptodate
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 78 bytes  received 136 bytes  428.00 bytes/sec
total size is 209,715,200  speedup is 979,977.57

Both files are reported as being up-to-date. But what happens if we unmount and remount the device and then sync?

$ umount /media/guyru/4E09-6457 
$ udisksctl mount --block-device /dev/disk/by-uuid/4E09-6457 
Mounted /dev/sdd1 at /media/guyru/4E09-6457.

$ time rsync -avv --update test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test-even is uptodate
test-odd
total: matches=0  hash_hits=0  false_alarms=0 data=104857600

sent 104,883,314 bytes  received 131 bytes  7,769,144.07 bytes/sec
total size is 209,715,200  speedup is 2.00

This time test-odd was reported as needing to be updated. So what changed? My guess is that while the device was mounted, modifcation dates were cached using normal granularity. Once I’ve remounted the file-system, the cache was purged, the the low-granularity modification date was used. That meant test-odd now had a modificaiton time which was one second before the original.

So how can it be avoided? Probably the trivial solution is to tell rsync not to copy over the modification date. This means that the copied files would probably anyway get a new modification time which is more than 1 second later than the original, hence even when rounding the modification time as FAT does it will still work. But a much better solution is built right into rsync: --modify-window

When comparing two timestamps, rsync treats the timestamps as
being equal if they differ by no more than the modify-window
value. This is normally 0 (for an exact match), but you may
find it useful to set this to a larger value in some situations.
In particular, when transferring to or from an MS Windows FAT
filesystem (which represents times with a 2-second resolution),
–modify-window=1 is useful (allowing times to differ by up to 1
second).

By using --modify-window=1 we can overcome the problem, as the files will be deemed as modified at the same time. Indeed it solves the problem:

$ rsync -avv --update --modify-window=1 test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test-even is uptodate
test-odd is uptodate
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 82 bytes  received 140 bytes  444.00 bytes/sec
total size is 209,715,200  speedup is 944,663.06

Leave a Reply

Your email address will not be published. Required fields are marked *

 

This site uses Akismet to reduce spam. Learn how your comment data is processed.