Re: [rfc] hide rs_buffers_t from public interface

--WIyZ46R2i8wDzkSu
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 18 Feb 2004, Donovan Baarda <[email protected]> wrote:
> In any case, flush is potentialy useful for different types of flush,
> as used in zlib. I haven't yet hit a case with librsync where I have
> really needed this, but I have used it a fair bit in zlib, so I can
> imagine a need for it. I probably prefer a seperate flush method, but
> guess a flush parameter would be more consistant.

That would be good.  I think that's fairly orthogonal to this
discussion?

> > > 3) It allows the application to be in full control of memory buffers
> > > to control and optimize memory usage.
> >=20
> > The library needs to do some dynamic allocation, so the application is
> > not strictly in complete control.
>=20
> Yeah. but minimizing how much it allocates helps for those in
> constrained memory land. Once again, not a biggie.

Is anyone actually going to use it in a space where allocating 10s of
kB of buffers is a problem?  There are not many such machines around
anymore.

Presumably the hash lookups are going to be a bigger deal.

> Nooo, not larger buffers.... :-)
>=20
> The problem is, as it is currently implemented, you nearly always end
> up with less than a complete block left in the internal buffer, so you
> always have to copy into the internal buffer. The _only_ time you
> don't is when the end of a block match happens to coincide exactly
> with the end of an input buffer. As you walk the block through the
> data, it needs to be contiguous, so you end up copying everything into
> the internal buffer just to keep the block together.
>=20
> If it was implemented right, you would only need a single fixed
> internal buffer of exactly block_len size. You don't really need data
> to be contiguous to calculate the block sums, so you can just walk
> over the fragment in the internal buffer, and onto the supplied input
> buffer. When you reach the end of the input buffer, you only need to
> copy the last non-matching block or block fragment into the internal
> buffer for next time.

Yes.  I think this can be fixed just in scoop.c?

> By changing the API to always require a minimum of block_len
> "lookahead" data on the input buffer, you can eliminate the input
> buffer entirely. It then becomes the applications responsibility to
> put the fragment left on the end of the input buffer onto the
> beginning of the next input buffer with the extra data.

Another approach would be to make it behave well if the application
gives us a lot of readahead, but not require it.

When the input buffer has insufficient readahead, we should *first*
return to the application and see if it can fill out the buffer a bit
more.  avail_in will be blocklen-1, so the application has to preserve
this data.  If the application still doesn't give us any more, then we
need to start accumulating data into an internal buffer.

> That last paragraph convinces me we should keep the internal buffer :-)
>=20
> > I thought about this a long time ago, and it seemed better to let the
> > application feed in single bytes if it wanted, even though that
> > required scoop.c.  Remember the application may not know the block
> > size, and there might be something about the protocol for the
> > application that makes it hard to build up whole blocks.
>=20
> Yeah :-)
>=20
> > > Even if we don't make this subtle change, the existing API
> > > implementation can be significantly simplified and optimized. It is
> > > possible to process nearly all data directly off the supplied input
> > > buffers, and only copy the less-than-block-size leftover fragments to
> > > a single fixed block_len sized internal buffer. The trick is you don't
> > > need to assemble contiguous blocks for calculating the sums... you can
> > > do it using seperate "sumUpdate" calls for the fragment on the
> > > internal buffer, and the rest on the input buffer.
> >=20
> > Right.
> >=20
> > Even before doing that, I think scoop.c can avoid shuffling data
> > around in many cases by using a larger buffer as I mentioned above.
> > I'm not sure if this is significant in the profile.
>=20
> Nah :-)=20
>=20
> bigger buffers won't help.=20

Uh, I think they would avoid shuffling data around.

Imagine if we had an infinite-size buffer.  We could just always
append to it, and never need to shuffle data back to the beginning.
(We could, if we wanted, reset to the beginning when the buffer
happens to be empty.)  The larger the buffer is, the more it
approximates that case.

--=20
Martin=20

--WIyZ46R2i8wDzkSu
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFAMrX5PGPKP6Cz6IsRAh4JAJ9tyqFoLkx8gSGuDMd+co1kHxgPXACglQZx
efbTEo2GTYXdrNRXJnLP7BQ=
=mVok
-----END PGP SIGNATURE-----

--WIyZ46R2i8wDzkSu--


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click