2 bugs from Billy on hdup

Billy Newsom <[email protected]> Mon, 21 Mar 2005 16:29:29 -0600
Newsgroups gmane.comp.sysutils.backup.hdup.general
Message-ID <[email protected]>
Well, so far, I had a few problems here.  I narrowed it down to some 
bugs in the code for one, and I'm not sure about the other.

First, when I used the "exclude =" line in the global section, I needed 
to empty that in some of my host sections.  But I found out the hard way 
that this causes the backup to miss a lot of files!

[global]
# don't include theses directories
exclude = lost\+found/, /proc/, /dev/, /sys/

[myhost1]
dir = /usr
exclude = ^/usr/ports, ^/usr/src, ^/usr/obj, ^/usr/X11R6

[myhost2]
dir = /boot, /root, /etc, /bin, /sbin, /lib, /libexec
exclude =

In the example above, myhost2 went crazy.  I had this in two host 
sections like that, and they both backed up around 60% of what they 
should have, with no errors.  So when I went to look through the tar.gz 
files, I happened to find out that a lot of files were missing.  What 
tipped me off also was that doing a debug mode showed a weird exclude 
line for myhost2:

Here is the actual debug output.  Look where the exclude pattern is 
"/boot" when I never said that in the config file!!
# hdup -D -V -V monthly ibm-root
hdup: No -c switch using default configfile: /usr/local/etc/hdup/hdup.conf.
hdup: ibm-root: STARTING BACKUP.
hdup: Overwriting old archive.
hdup: Creating LOCK.
hdup: Using inclist /dellbak/ibm-root/etc/inclist.monthly.
[DEBUG] exclude pattern: [/boot]
[DEBUG] include pattern: [.*]
[DEBUG] DE 0 FE 1 DI 0 FI 1

[DEBUG] Walking: /libexec
hdup: Files found: 2.
[DEBUG] Walking: /lib
hdup: Files found: 32.
[DEBUG] Walking: /sbin
hdup: Files found: 155.
[DEBUG] Walking: /bin
hdup: Files found: 196.
[DEBUG] Walking: /etc
hdup: Files found: 534.
[DEBUG] Walking: /root
hdup: Files found: 837.
[DEBUG] Walking: /boot
hdup: Files found: 837.
******snip*************
Archive size.:  18.5M
Elapsed......:  0:01:02
Status.......:  successfully performed backup

When I removed all instances of the exclude line from the config file, I 
got better results, and /boot was backed up.  Notice the much larger 
archive and double the execution time.

ibm:/bin# hdup -D -V -V monthly ibm-root
hdup: No -c switch using default configfile: /usr/local/etc/hdup/hdup.conf.
hdup: ibm-root: STARTING BACKUP.
hdup: Overwriting old archive.
hdup: Creating LOCK.
hdup: Using inclist /dellbak/ibm-root/etc/inclist.monthly.
[DEBUG] include pattern: [.*]
[DEBUG] DE 0 FE 0 DI 0 FI 1

[DEBUG] Walking: /libexec
hdup: Files found: 2.
[DEBUG] Walking: /lib
hdup: Files found: 32.
[DEBUG] Walking: /sbin
hdup: Files found: 155.
[DEBUG] Walking: /bin
hdup: Files found: 196.
[DEBUG] Walking: /etc
hdup: Files found: 535.
[DEBUG] Walking: /root
hdup: Files found: 838.
[DEBUG] Walking: /boot
hdup: Files found: 2029.
******snip*************
Archive size.:  42.5M
Elapsed......:  0:02:44
Status.......:  successfully performed backup


The second problem that I can't figure out is that this archive above is 
either corrupted or is being aborted somehow in the middle.  I never get 
all of the files in /root, only a certain few (all happen to start with 
a period (.).  WinRAR says that it is corrupted.

I did some testing on this, because I can tell where the file was ended. 
  It appears that when a certain subdirectory of /root is backed up 
(called .spamassassin), it will fail to some extent.  WinRAR reports 
corruption.  gzip says the file tests okay.  It looks to me that there 
is probably a file here which is changing as the backup is occuring.

gtar says that there are some file differences:
#gtar -dzf /dellbak/ibm-root/2005-03-21/ibm-root.2005-03-21.monthly.tar.gz
root/.spamassassin/auto-whitelist: Mod time differs
root/.spamassassin/auto-whitelist: Contents differ
gtar: Skipping to next header
gtar: Archive contains obsolescent base-64 headers
root/.spamassassin/auto-whitelist.mutex: Mod time differs
root/.spamassassin/bayes.mutex: Mod time differs
root/.spamassassin/bayes_journal: Mod time differs
root/.spamassassin/bayes_journal: Size differs
root/.spamassassin/bayes_seen: Mod time differs
root/.spamassassin/bayes_seen: Contents differ
gtar: Skipping to next header
root/.spamassassin/bayes_toks: Mod time differs
root/.spamassassin/bayes_toks: Contents differ
gtar: Skipping to next header
gtar: Error exit delayed from previous errors

Those are the files which seem to be the corrupting factors in the 
eventual tar.gz file.  It looks to me that gtar might be the culprit 
here -- it somehow doesn't get file piped into gzip before the file is 
changed, I guess.

Do you know what the "obsolescent" headers are?

The current workaround is to not backup this entire directory, but I 
would like to include it if possible.

Thanks,
Billy Newsom