announcing rsyslog 3.11.0 - first open source disk buffering syslogd

[I am the developer of rsyslog and as such have a certain bias. I still
think this mail is technically important and thus I am posting to this
list (pending moderator decision, of course).]

Hi all,

it is a great pleasure for me to do a fully buzzword-compliant ;)
announcement for rsyslog 3.11.0, which is available immediately. This
release is a very important milestone:

Now rsyslog is the world's first-ever purely open-sourced syslogd
supporting disk based buffering individually for each log destination[*]
(called "action" in rsyslog speak). Besides, each action can now be
executed on its own thread. Together with the multi-threaded main worker
pool, rsyslog is more than ready for tomorrow's massively multi-core
machines. I think another "world's first" is the ability to
automatically discard low priority syslog messages under congestion
conditions.

The new engine provides, among others:

* reliable storage of messages while a destination is down
* persistence of message even when rsyslogd is shut down
* automatic retry of failed actions until they succeed,
  of course without any message loss
* unparalleled ability to handle message burst
* fully automatic worker thread pools
* slow actions (e.g. database writes) are de-coupled from
  quick local actions (e.g. file writes)
* rate-limited action processing
* flow control slows senders down when queues get full
* ability to discard less important messages in favor
  of higher important ones when the queue runs out of
  space 

All of this while using minimal system resources.

The system can be set up with few configuration commands, but provides
very in-depth commands to manipulate aspects like worker thread startup
and shutdown parameters, queue sizes, disk quotas, watermark algorithm
settings and many more (sometimes exotic) tweaking knobs.

Full details about the new system and its parameters can be found at:
   http://www.rsyslog.com/doc-queues.html

An actual use case, doing massive database inserts, is documented here:
   http://www.rsyslog.com/doc-rsyslog_high_database_rate.html

Rsyslog 3.11.0 is fresh off the "development press". While it is well
tested in lab, I expect that there are bugs, especially given the
magnitude of changes and the massively multithreaded processing flow. As
such, I do not yet advise to use it on important production machines.

I would, however, be very interested in feedback from the field,
including bug reports. My focus in the next weeks will be on making the
new engine rock-solid while at the same time introducing many new
features which are quite easy to build on top of the new engine (like
more advanced rate limiting algorithms, action execution-time windows
and more advanced ways to flow-control inputs). My hope is that within
two month or so we will have a very stable version suitable for
production use in the most demanding environments.

I hope that this announcement will motivate some people to try rsyslog
out and provide feedback to the project. Just be warned that I will not
be at my desk next week: I am taking my now hopefully deserved break
after a long time of hard coding. I thought to delay the announcement,
but have decided it is probably better to get some head start. So please
do not be disappointed if I do not reply immediately on comments posted.
For comments and questions, I recommend using either the mailing list or
the web forum at

http://www.rsyslog.com

You may download the tarball either from the rsyslog site above or from
its Freshmeat info page:

http://freshmeat.net/projects/rsyslog/

Rsyslog is released under GPLv3. Those interested in new release
notifications are advised to subscribe to rsyslog on Freshmeat. That
service provides low-volume update notifications (in contrast to the
mailing list, which can be quite bursty at times).

Finally, I would like to thank everybody who contributed to the project.
Without contributions (of any type), we would definitely not be able to
achieve this important milestone. Please keep the good stuff coming!

I hope rsyslog will be useful. All feedback is highly appreciated.

Best regards,
Rainer Gerhards
Adiscon


[*] I know that syslog-ng does disk buffering, but it does not support
it in the open source edition.