comparison of star and rsync methods

--===============1282674745==
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="6zdv2QT/q3FMhpsV"
Content-Disposition: inline


--6zdv2QT/q3FMhpsV
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

* Message by -Joerg Schilling- from Tue 2008-03-11:
> Lasse Kliemann <[email protected]> wrote:
>=20
> > But I tried something else, maybe it's interesting:
> >
> > - Filesystem A is busy, take snapshot S from it.
> > - Synchronize filesystem B with S using 'rsync -axH --inplace'.
> > - Remount B read-only.
> > - Take an incremental dump from B.
> > - Restore the incremental dump(s) taken this way to a filesystem C.
> > - Compare the contents of S with C using diffopts=3D\!ctime,\!lmtime,\!=
atime.
> > - Release S.
> > - Start over after some time.
> >
> > So, in B we keep a kind of "mirror" of A, which has the advantage that =
we can=20
> > mount it read-only and that it is not a snapshot. B is not changed betw=
een=20
> > iterations; all we do with it is one run of rsync before each dump. (Wi=
th=20
> > rsync 3.0.0 a patch [1] is suggested for this.)
>=20
> Could you tell us how the speed of a rsync call and an incremental star=
=20
> dump/restore compare? Which is faster and what is the difference?

Here are some figures.

#  |  rsync S --> V  |  rsync S --> B  |  star B  |  star S  |  approx. siz=
e | ratio
---------------------------------------------------------------------------=
----
0  |             85  |             54  |      33  |      57  |    961640.00=
k | .67
1  |            600  |            593  |     217  |     349  |   2485870.00=
k | .58
2  |             10  |             86  |       1< |       1< |        40.00=
k |  -
3  |            765  |            813  |     297  |     439  |   3328160.00=
k | .57
4  |              2  |            109  |       1< |       6  |        20.00=
k |  -
5  |            743  |            770  |     289  |     428  |   3329230.00=
k | .58

All times given in seconds. The previous to last column gives how much data=
=20
had to be transfered (by star). One row corresponds to one run of each meth=
od=20
on the same filesystem snapshot S. All filesystems are ext3 on encrypted=20
devices. The snapshot S is on a RAID-1, and all other filesystem are on a=
=20
RAID-0. The last column gives the ratio of the methods 'rsync S --> V' and=
=20
'star S' (where it makes sense to compute it).

Note: maybe these figures will change if one of the parameters is changed,=
=20
e.g., no encryption, different filesystem, etc. I have not investigated thi=
s,=20
since it would require a lot more work.

For runs 0 to 3, the methods were invoked in the order given, from left to=
=20
right. For subsequent runs, the order was kind of reversed: 'star S', then=
=20
'rsync S --> B', then 'star B', finally 'rsync S --> V'. Why this order was=
=20
chosen (and not a full reverse order), is that 'star B' depends on 'star S=
=20
--> B'. This will become clear by an explanation of the methods.


METHODS

The different methods are:

rsync S --> V: the snapshot S is kind of copied into an empty directory,=20
making hardlinks to files of a previous such run where appropriate and only=
=20
transfering changed files. This uses rsyncs --link-dest option. This method=
=20
is applicable as a backup system of its own. The complete call that I use i=
s:
   rsync -axH --inplace --link-dest=3D"$PREVIOUS" "$SOURCE"/ "$DEST"
What is $DEST in this run will become $PREVIOUS in the next run.
This method gives copies that are identical in the sense of
   diffopts=3D\!ctime,\!lmtime,\!atime,\!nlink=20
I've tested this for some time now.

rsync S --> B: the filesystem B is synchronized with S using 'rsync -axH=20
--del --inplace'.

star B: an incremental dump is taken of filesystem B, after B it has been=
=20
synchronized with S with the method above.

star S: an incremental dump is taken of the filesystem snapshot S.


RUNS

Run 0 is the initial one, meaning there were no previous copies or dumps on=
=20
the receiving side, in particular meaning that level 0 dumps were done by=
=20
star.

Run 1 was done after the creation of a many small files (using blogbench).=
=20

Run 2 was done after all these files were deleted. Method 'rsync S --> V' i=
s=20
much faster than 'rsync S --> B', which can be explained by the fact that t=
he=20
second method had to delete a lot of files on B.

Run 3 was again done after the creation of many small files.

Run 4 (now in "reverse" order of the methods) was done after all the small=
=20
files were deleted.

Run 5 was again done after the creation of many small files.


DISCUSSION

Doing 'star S --> B' and then 'star B' is a backup method, but it has a lot=
=20
of ovehead. It was invented for testing only, because I had problems with t=
he=20
'star S' method. (Differences after restore; I explained that in the other=
=20
messages. This combined method, however, never gave differences.)

So the real competitors are 'rsync S --> V' and 'star S' (hoping that the=
=20
problems with the latter can be resolved). Clearly, star is faster than=20
rsync; the star runs only take about 60% of the time of the rsync runs. The=
=20
big structural difference is that with rsync, one has the backup in a=20
filesystem instead of in a single file (or in few files with incremental=20
dumps). This can be an advantage if one often has to access particular file=
s=20
in the backup. It can also be a disadvantage if one wishes to write the=20
backup to some external medium or push it over the network.


--6zdv2QT/q3FMhpsV
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)

iQIcBAEBAgAGBQJH2q22AAoJEFYll6N4nhdvYFoQALXGUsyKrBZW3q16aNjWLadQ
19vgZXuFr2Ysj7OfiSkL9E8GrMqQUgIFCcuOIkkioau1I2zEApdIyIVIa7HPlnHL
P+bx8oIWdDKCCR5tPC3SHD45n/ldBrj74Lm7CcTN/v7O9b63I0zkLWqtEvYxQZN3
Oo3iS63s7D7oa+uWIyt3KJJeJXFAUE0upwZEIwzHsJtXEqwVMdLDHdC3is4Q/Z7U
BflhpaTiABcUqrbnzJoeOU5y1o7p2NUK30zKua0doF49RB0cBQzfNsRDbSZwqjkV
LhzJX5MjPgodZKVpuxaOJT10vUL4QoEFV4B1v/cJnc88/mC7rfeoJq4glyAXRO4K
Fz9nSWy81h4H2MkcZwO5An404ACUxX+2BX/urWMn6pJD4U7q8nueR3qLvjo1bJuH
ntaIMLFwlGlklyjJOfcuHQEnGvEKwXfKvWybOrXXhshEGGmBgRhtmzS5bYcrzIr9
qKXx2caFAnE3so+hsoXJAyb8mbjIJGQbhFIQLJF9C+HEw3rJqr+ju3WxMH2LWStR
jZA0vkhBiltCjIeMX8o+Q98ySVmr4nclC6MqbpBTSwZ1CRLm7iGmA/YkX4U/cMNb
WfGrd5p3h+fiGsVHy1bwD7FGxNdnuzJnNV1NtumAvokGSmgfbcvPc7RbjzCycymU
7pV+ES1EWJfXhgvQ4m/C
=fEVK
-----END PGP SIGNATURE-----

--6zdv2QT/q3FMhpsV--

--===============1282674745==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Star-users mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/star-users

--===============1282674745==--