Re: burning changing files

This is getting off the K3b topic, but I thought I'd share my followup findings a bit.  It's possible to stream a tar to an ISO using growsiofs --stream-media-size option (see mkisofs man page).  I have problems with this with large files... not sure why yet.. maybe filesize limit.  With small files this allowed me to mount the disc and navigate through the tar with tar capabale browsers, specifically Konqueror.  

You can also pipe the tar directly to the dvd.  Konqueror can read the tar by opening the device link directly, but for some reason it breaks.  It usually only shows me one or two directories in the tar when there are many.  I've tried with star too, same result.  However, I do get tars that work even for big files... they just have to copied or extracted to use them... better than corrupted data.

Now all I need is a good verification method.  You can use the -d switch in tar to verify after the fact.  However, it only verifies files that are in the tar, not ones that never made it there (which I've had happen).  What I decided to do is construct the MD5 as the tar is being piped to disk and then check against the disk file later.  This only checks the burn, not the read, but I'm not usually worried about my systems read stability, could be an issue if you have bad ram or a failing drive though so this verification isn't for a purist.  To me the most important thing is for backup to be easy.  If I back up often, then even if I miss the bad hd symptoms... I'll have data often enough not to lose much.  Maybe I should worry about tar/star corrupting data though.. this won't get caught either.

This method has many advantages.  It doesn't have to reread the data from the hard disk, so it's faster and doesn't keep tying up your hd while you're working.  I'm trying to avoid temporary disk space so I don't have a temporary copy of the tar and even if I remade one from disk, the point of my problem is that files on disk are changing... so really to check against the disk files, the only way is to check md5's one file at a time.  This is VERY slow even off a standard ISO image, and when comparing to a tar, not so easy to even see how to do without extracting the whole tar first (extracting one file at a time is impossibly slow)... and thus defeating all the points.

There's a problem though.  The optical device data stream doesn't end exactly where the tar pipe ended so an MD5 sum comparisson won't match up... unless you dd the correct number of bytes off the device... so you have to pay attention while piping the tar to disk and somehow get the exact byte length.  dd outputs this, but I must admit my ignorance that I don't know how to intercept it, because the output is not standard output and is not standard error either.  Instead I used star's standard error reporting (too bad since using dd would allow choice of archiver... I'll work on it).    Finally, you get a simple pass or fail.  Even if you did compare files one at a time after the burn, you'd have to check by eye that the ones that did change are one's you would expect to change.  

here's the scriptlet:

# The tar will pipe to growisofs, but will also tee to a named pipe that will be used
# to read the md5.. Make the named pipe:
rm tfifo
mkfifo tfifo

# first start md5sum to read off the named pipe
# also dump the star standard error ouput to disk so we can use the byte length info from it
md5sum -b tfifo  > md5original & star -c $@ 2> tarfilesize | tee tfifo \
|growisofs  -Z $dev=/dev/fd/0

#parse the star output to get the byte length; toolazy to remember gawk right now.
length=`head -n 1 tarfilesize|cut -d"=" -f2|cut -d"." -f1|cut -d" " -f2`
echo file length $length kB
rm tarfilesize

#now read back the right number of bytes from the burned disk to check the MD5
dd if=$dev bs=1024 count=$length |md5sum >md5copy

echo doing MD5 check:
mdo=`head -n 1 md5original|cut -d" " -f1`
echo "original: $mdo"
mdc=`head -n 1 md5copy|cut -d" " -f1`
echo "copy: $mdc"
echo
#compare the disk md5 to the tar pipe:
if [[ $mdo == $mdc ]]; then
echo passed
else
echo failed
fi
rm tfifo

_________________________________________________________________
Live Earth is coming.  Learn more about the hottest summer event - only on MSN.
http://liveearth.msn.com?source=msntaglineliveearthwlm
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/