I’m using rsync to sync files from my computer to a FAT-formatted SD card. Using the --update flag with rsync makes it “skip files that are newer on the receiver.” It seems that it should work as follows: after the first sync, any subsequent syncs will only transfer those files that changed in the meantime. However, I noticed that transfer times are usually longer than expected, which led me to think that things are not working as they should.
As --update relies only on the modification date, obviously something is wrong with it. After ruling out several other possibilities, I guessed that the modification date of some files gets mangled. A quick check of FAT revealed that FAT can only save modification times with 2-second granularity. I’ve also used rsync’s --archive flag, which, among other things, attempts to preserve the modification times of the files it copies. But what about FAT’s lower granularity for modification time? That was apparently the culprit. Some files, when copied, got a modification time that was up to 1 second before the original modification time. Hence, every time I synced, from rsync’s perspective, the target was older than the source and therefore needed to be overwritten.
This can be demonstrated by the following session:
$ dd if=/dev/urandom of=test-even bs=1M count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 6.20716 s, 16.9 MB/s $ cp test-even test-odd $ touch -t 201701180000.00 test-even $ touch -t 201701180000.01 test-odd $ stat -c %Y test-* 1484690400 1484690401
We’ve created two files, test-even and test-odd, with random content. We’ve set the modification time of test-even to be on an even second, and the modification time of test-odd to be on an odd second (one second apart). Simply syncing the files to a FAT-partitioned device works as expected:
$ rsync -av --update test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
test-even
test-odd
sent 209,766,550 bytes received 54 bytes 7,627,876.51 bytes/sec
total size is 209,715,200 speedup is 1.00
Resyncing immediately also works:
$ rsync -avv --update test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test-even is uptodate
test-odd is uptodate
total: matches=0 hash_hits=0 false_alarms=0 data=0
sent 78 bytes received 136 bytes 428.00 bytes/sec
total size is 209,715,200 speedup is 979,977.57
Both files are reported as being up to date. But what happens if we unmount and remount the device and then sync?
$ umount /media/guyru/4E09-6457
$ udisksctl mount --block-device /dev/disk/by-uuid/4E09-6457
Mounted /dev/sdd1 at /media/guyru/4E09-6457.
$ time rsync -avv --update test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test-even is uptodate
test-odd
total: matches=0 hash_hits=0 false_alarms=0 data=104857600
sent 104,883,314 bytes received 131 bytes 7,769,144.07 bytes/sec
total size is 209,715,200 speedup is 2.00
This time test-odd was reported as needing to be updated. So what changed? My guess is that while the device was mounted, modification dates were cached using normal granularity. Once I remounted the file system, the cache was purged, and the low-granularity modification date was used. That meant test-odd now had a modification time that was one second before the original.
So how can it be avoided? Probably the trivial solution is to tell rsync not to copy over the modification date. This means that the copied files would probably get a new modification time that is more than 1 second later than the original, so even when FAT rounds the modification time, it will still work. But a much better solution is built right into rsync: --modify-window
When comparing two timestamps, rsync treats the timestamps as
being equal if they differ by no more than the modify-window
value. This is normally 0 (for an exact match), but you may
find it useful to set this to a larger value in some situations.
In particular, when transferring to or from an MS Windows FAT
filesystem (which represents times with a 2-second resolution),
–modify-window=1 is useful (allowing times to differ by up to 1
second).
By using --modify-window=1, we can overcome the problem, as the files will be deemed to have been modified at the same time. Indeed, it solves the problem:
$ rsync -avv --update --modify-window=1 test-{even,odd} /media/guyru/4E09-6457
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
test-even is uptodate
test-odd is uptodate
total: matches=0 hash_hits=0 false_alarms=0 data=0
sent 82 bytes received 140 bytes 444.00 bytes/sec
total size is 209,715,200 speedup is 944,663.06