Re: New ftpsync version 20160306

On Sun, 06 Mar 2016, Joerg Jaspert wrote:

> - trace files changed format and include more data

The additional data in trace files now includes
 - the start and end date of the mirror run in proper RFC822 format,
 - the "archive version", as taken from the trace/master file,
 - information about the individual syncing stages (time, traffic), and
 - whether SSL was used to connect to the upstream rsync service.

e.g.  https://syncproxy2.eu.debian.org/debian/project/trace/syncproxy2.eu.debian.org

> - ftpsync tries to store the hierarchy using new _<foo> files

Here, we have two new (sets of) files in the trace directory.

The first, _traces, is just an ls of the trace files, sorted by mtime.
So, if all the mirrors in your chain have the correct system time (not
always the case) and if they only create a single tracefile after their
sync, then that should result in the complete path from the master to
this mirror being correctly reflected in the _traces file.  See, for
instance, https://syncproxy.au.debian.org/debian/project/trace/_traces
or http://debian.sil.at/debian/project/trace/_traces .

The other set is _hierarchy*, which attempts to collect the same
information even in the presence of clock skew.  However, it requires
that all mirrors in the path have syncscripts that support collecting
this information.  The way it works in conforming mirror scripts is that
_hierarchy is local only, not overwritten or touched by rsync in any
way.  A mirror run however copies _hierarchy.mirror from the upstream
server.  After the mirror run, information is appended to
_hierarchy.mirror and that makes the content of the new local _hierarchy
and _hierarchy.mirror files.

Consumers of this file should download _hierarchy, as that is intended
to stay correct even while the server in question updates.  The
_hierarchy.mirror file instead may contain the upstream server's
information while the script runs.


> - rsync over ssl support (this needs extra tools and a server that
>   supports it)

By setting RSYNC_SSL to true in ftpsync.conf, ftpsync will try to
connect to the upstream server using rsync over SSL.  We have picked
port 1873 by default, but this can be changed in the config.
Currently, we support two methods for wrapping rsync in SSL, socat and
stunnel.  Both means will verify that the server presents a certificate
issued by a CA present in /etc/ssl/certs (override with
RSYNC_SSL_CAPATH).

 - socat (RSYNC_SSL_METHOD=socat) will only do name-checks starting with
   1.7.3, available in jessie-backports.  Prior to that it will accept
   any cert issued by a valid CA regardless of whether or not it is the
   one you expected.
 - stunnel4 in stable likewise does not support name checks.  To use
   this version of stunnel, set RSYNC_SSL_METHOD=stunnel4-old.  Starting
   with stretch (jessie+1), stunnel will support name checks.
   Backporting this version is tricky, as it also requires openssl 1.0.2
   (stable has 1.0.1).

(The default is stunnel4).

Note that diagnostics are somewhat lacking, so debugging issues could be
harder than it should be.

Useful debugging steps might include running rsync-ssl-tunnel in
archvsync directly.  You should see the rsync prompt.
} $ RSYNC_SSL_METHOD=socat ./bin/rsync-ssl-tunnel syncproxy2.eu.debian.org
} @RSYNCD: 31.0

If that works, try wrapping it in an rsync call:
} $ RSYNC_SSL_METHOD=socat rsync -e ./bin/rsync-ssl-tunnel syncproxy2.eu.debian.org::
} debian          Full Debian FTP Archive (contact [email protected] for access; see https://www.debian.org/mirror/size for size)
} debian-debug    Debug packages.  Probably large.  Starting end of 2015.

Some of the debian.org syncproxies now have SSL-enabled rsync ports.
Those are syncproxy2.eu, syncproxy3.eu, syncproxy2.wna, syncproxy.cna,
and syncproxy.au.

Cheers,
weasel