Re: Alternative to rrdcached on linux - rrdfs - FUSE based RRD caching

Hi Peter,

wow ... sounds impressive. Have you done performance evaluations ?

If you could apply your genius to rrdcached and maybe make it even
better based on the insights gained from the rrdfs implementation?

cheers
tobi



Today Peter Stamfest wrote:

> Hello!
>
> In the last couple of days, I implemented an alternative to the rrdcached:
> A FUSE based linux file system doing essentially the same as rrdcached.
>
> The idea is as follows:
>
> - There is a directory hierarchy (the permanent storage area), that holds
> RRD files.
> - Another directory hierarchy is used to keep cached RRD files, I usually
> put this below /dev/shm (that is, into a memory based filesystem for
> extremely fast access), I call this the staging area.
> - The rrdfs file system provides a layer above these directories,
> providing the same view as in permanent storage
> - Updating RRDs is done by writing data to a special INPUT file in the
> format
>
>   FILENAME <SPACE> <timestamp>:<ds-value>:<ds-value>:....
>
> - Alternatively, data can be appended to the RRD file presented over
> rrdfs. In that case the FILENAME must not be written.
> - A read access over rrdfs causes data to be updated to the staging area
> before handing RRD data to the reader
> - Unlinking the RRD over rrdfs causes the staged data to be transferred
> back to permanent storage
> - An optional cleanup thread causes the RRD updating and transfer from
> staging to permanent storage from time to time.
> - plus a log-file is written in a write-ahead fashion. The log file gets
> rolled automatically by the cleanup thread.
>
> I started the implementation only about a week ago, without knowing about
> rrdcached, so I was pretty embarassed when I found out about it. But I
> don't want this to be a complete waste of time.
>
> By the way: The thing is fully multithreaded.
>
> So far I only tested it in a 1.2.x environment, 1.3 and 1.4 pending. The
> ONLY RRD API used is rrd_update_r, so I'm pretty sure there are no real
> hard problems to solve to get it to work with newer versions.
>
> One nice feature is the immediate availability of RRD to any programming
> language that can append to files: no need to start a new process.
>
> Downloads:
>
> http://oss.stamfest.net/rrdfs/rrdfs-1.0.tar.gz
>
> Mercurial repo available on request.
>
> I would be willing to actually have this distributed alongside of rrdtool
> proper. But as I said: It is an alternative to rrdcached and it only works
> on linux. I'd be happy to actually hear thoughts about the usefulness of
> this beast. Benchmarking results would also be great.
>
> peter
>
>
> _________________________________________________________________________
> Dipl.-Ing. Peter Stamfest     UNIX, Networking & Computing Consultant
> Tel: +43/699/10711205         Software Development - Internetservices
> E-Mail: [email protected]     WWW: http://stamfest.at/
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch [email protected] ++41 62 775 9902 / sb: -9900