So when Jon Flowers asked in the comments of the original tarsum post about the possibility of getting the checksums of files in the tar file without extracting all the archive, I’ve decided to re-tackle the problem.

This time I’ve chose python and by using the tarfile and hashlib modules I came up with a solution that allowed me to go over tar files to calculate the checksum values without extracting all of them to the disk. However some sacrifices where made in the form of back-compatibility of the output. I’ve tried to make the interface similar to the old one, and have kept all the command line options. Instead of specifying a program name to calculate the checksum values (such as sha1sum) as argument to --checksum you specify the name of the checksum algorithm such as md5, sha1, sha256, sha512 (or any other supported by hashlib).

Other changes where made so tar files can be piped directly into tarsum (which also works transparently with bzip2 and gzip compression).

tarsum < sometarfile.tar.gz > sometarfile.tar.gz.md5

Performance-wise, according to some tests I’ve carried out, the new version is faster with big tar files than the old one, but it’s the other way around with small archives (which I find less important).

#! /usr/bin/env python
 
# Copyright (C) 2008-2009 by Guy Rutenberg
 
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the
# Free Software Foundation, Inc.,
# 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
 
import hashlib
import tarfile
 
def tarsum(input_file, hash, output_file):
        """
        input_file  - A FILE object to read the tar file from.
        hash - The name of the hash to use. Must be supported by hashlib.
        output_file - A FILE to write the computed signatures to.
        """
        tar = tarfile.open(mode="r|*", fileobj=input_file)
 
        chunk_size = 100*1024
        store_digests = {}
 
        for member in tar:
            if not member.isfile():
                continue
            f = tar.extractfile(member)
            h = hashlib.new(hash)
            data = f.read(chunk_size)
            while data:
                h.update(data)
                data = f.read(chunk_size)
            output_file.write("%s  %s\n" % (h.hexdigest(), member.name))
 
def main():
    parser = OptionParser()
 
    version=("%prog 0.2.1\n"
             "Copyright (C) 2008-2009 Guy Rutenberg <http://www.guyrutenberg.com/contact-me>")
    usage=("%prog [options] TARFILE\n"
           "Print a checksum signature for every file in TARFILE.\n"
           "With no FILE, or when FILE is -, read standard input.")
    parser = OptionParser(usage=usage, version=version)
    parser.add_option("-c", "--checksum", dest="checksum", type="string",
        help="use HASH as for caclculating the checksums. [default: %default]", metavar="HASH",
        default="md5")
    parser.add_option("-o", "--output", dest="output", type="string",
        help="save signatures to FILE.", metavar="FILE")
 
    (option, args) = parser.parse_args()
 
    output_file = sys.stdout
    if option.output:
        output_file = open(option.output, "w")
 
    input_file = sys.stdin
    if len(args)==1 and args[0]!="-":
        input_file = open(args[0], "r")
 
    tarsum(input_file, option.checksum, output_file)
 
if __name__ == "__main__":
    from optparse import OptionParser
    import sys
    main()
 
# vim: ai ts=4 sts=4 et sw=4

You can get a wget’able version here: tarsum-0.2.bz2.

Update 2009-08-12: Removed excess argument to tarsum() and switched the filemode to r|* (from r:*). Bumped version string.

Some time ago, I got back a hard disk back from data recovery. One of the annoying issues I encountered with the recovered data was corrupted files. Some files looked like they were recovered successfully but their content was corrupted. The ones that were configuration files, where usually easy to detect, as it raised errors in programs that tried to use them. But when such error occurs in some general text file, (or inside the data of an SQL dump), the file may seem correctly fine unless closely inspected.

I have an habit of storing old backups on CDs (they are initially made to online storage), I do it in order to reduce backup costs. But the recovered/corrupted data issue raised some concerns about my ability to recover using this disks. Assuming that I have a disk failure, and I couldn’t recover from my online backups for reason, how can I check the integrity of my CD backups?

Only storing and comparing hash signature for the whole archive, is almost useless. It allows you to validate whether all the files are probably fine, but it can’t tell apart one corrupted file in the archive from a completed corrupted archive. My idea was to calculate checksum (hash) for each file in the data and store the signature in a way that would allow me to see which individual files are corrupted.

This is where tarsum comes to the rescue. As it’s name applies it calculate checksum for each file in the archive. You can download tarsum from here.

Using tarsum is pretty straight forward.

tarsum backup.tar > backup.tar.md5

Calculates the MD5 checksums of the files. You can specify other hashes as well, by passing a tool that calculates it (it must work like md5sum).

tarsum --checksum=sha256sum backup.tar > backup.tar.sha256

To verify the integrity of the files inside the archive we use the diff command:

tarsum backup.tar | diff backup.tar.md5 -

where backup.tar.md5 is the original signature file we created. This is possible because the signatures are sorted alphabetically by the file name inside the archive, so it the order of the files is always the same.

Note that if you use an updated version of GNU tar, tarsum can also operate directly on compressed archives (e.g. tar.bz2, tar.gz).