Italian mirror is now fully functional

Marco Nenciarini <[email protected]> Fri, 7 Jan 2005 17:31:20 +0100

Newsgroups	gmane.comp.web.htdig.devel
Message-ID	<[email protected]>

The italian mirror (www.it.htdig.org) now support full ftp and http
mirror (even patch and files) and is updated daily.

The new mirror script use puf to get files over http from mainsite,
and use the index.html files to remove old files.

The URIS are: 
http://www.it.htdig.org/files/
http://www.it.htdig.org/htdig-patches/
ftp://ftp.it.htdig.org/pub/htdig/files
ftp://ftp.it.htdig.org/pub/htdig/htdig-patches

The mirror script is attached.

-- 
---------------------------------------------------------------------
|    Marco Nenciarini    | Debian/GNU Linux Developer - Plug Member |
| [email protected] | http://www.prato.linux.it/~mnencia       |
---------------------------------------------------------------------
Key fingerprint = FED9 69C7 9E67 21F5 7D95  5270 6864 730D F095 E5E4

mirror (text/plain, 1.3 KB)

#!/bin/bash

PATH=/bin:/usr/bin:/usr/local/bin
BASEDIR=/org/www.it.htdig.org
WWWDIR=$BASEDIR/www
FTPDIR=$BASEDIR/ftp
FILEDIR=$FTPDIR/files
TMPDIR=$BASEDIR/tmp

set -e

# Update web
cd $WWWDIR
/usr/bin/cvs -q -z6 update -Pd

# Update files
cd $FTPDIR
puf -r -u -ns -xi .ht-update-in-progress.puf http://www.htdig.org/files > /dev/null

# Purge old files
listfiles() {
    test -r "$1" &&
    cat $1 | \
    sed -ne "s/\HREF=/\n%=/ig;
         s/%=\"\([^\"]*\)\"/%=\1\n/g;
	 /^[^%][^=]/D;
	 :loop;
	 s@^%=@$2@g;
	 T no;
	 P;
	 :no;
	 D;
	 b loop" | \
    grep -v "?\|^$2/"
}

rm -f $TMPDIR/filesok
exec 3> $TMPDIR/filesok
echo $FILEDIR >&3
newfiles=$(listfiles $FILEDIR/.ht-update-in-progress.puf $FILEDIR/)
while [ -n "$newfiles" ]; do
    files=$newfiles
    newfiles=
    for file in $files; do
        if [ -d $file ]; then
	    file=${file%/}
	    newfiles="$newfiles $(listfiles $file/.ht-update-in-progress.puf $file/)"
	fi
	echo $file >&3
    done
done
exec 3> /dev/null

rm -f $TMPDIR/filesok.sort $TMPDIR/files.sort
sort $TMPDIR/filesok > $TMPDIR/filesok.sort
find $FILEDIR | sort > $TMPDIR/files.sort

diff $TMPDIR/files.sort $TMPDIR/filesok.sort | sed -ne "s/^< \(.*\)$/\1/p" | xargs rm -fr

rm -f $TMPDIR/filesok.sort $TMPDIR/files.sort $TMPDIR/filesok

##Update patches
wget -q -nv -m -np -nH -p ftp://ftp.ccsf.org/htdig-patches
find . -name .listing -exec rm -f {} \;

signature.asc (application/pgp-signature, 189 B)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFB3rlYaGRzDfCV5eQRApS5AJ9ZRuNWXJo6djjjDy/2Pt8PjdPFQQCcD1HE
4zlNh6in2Sja5a8FZ7ocFZc=
=m/Lx
-----END PGP SIGNATURE-----