IFS=”
”
for line in $(cat ${md5file})
do
md5=$(echo ${line}|awk ‘{print $1}’)
filename=$(echo ${line}|awk ‘{print $2}’)
md5archivefile=$(tar -zxOvf myfile.tgz ${filename} 2>/dev/null | md5sum – | awk ‘{print $1}’)

if [ ! "${md5archivefile}" == "${md5}" ]; then
echo “NOT OK: $filename,$md5,$md5archivefile”
else
echo “OK: $filename,$md5,$md5archivefile”
fi
done

I admit that I’ve never tested the script with such a big file like you did. What suprised me that setting the filemode to r|* slowed the script a bit (at least for my ~300MMB tar), I assumed that giving up random access, should make things faster, but it didn’t

Anyway I’ve fixed both issues. Thanks again.

Tried out your program using Ubuntu 9.04, but I encountered two problems.

First, in the last line of main() you called tarsum() with 4 parameters while tarsum() is defined to accept only 3 parameters. Python is aborting the program because of this. I’m not a python programmer, but when I took out the 4th parameters, the program now runs.

Then I tried to use the program on a 21GB bzipped tarball that contains a 100GB file. While the program runs, it runs for less than 1 second. It also prints out a checksum that is different. The file is ok when tested using “bzip2 -t”.

After some research, I changed the filemode from “r:*” to “r|*” to use stream IO. After this change, “tarsum-0.2 file.tar.bz2″ now aborts, but “bunzip2 -c file.tar.bz2 | tarsum-0.2″ now seems to work.