Re: Database for greylisting plugin.

On Fri, Dec 08, 2006 at 05:17:08PM +1100, Graham Miller wrote:
> If it was designed to do basic greylisting (sender-receiver-ip) using the
> filesytem as a database (emtpy files and use mtime), then it cannot
> differentiate between a triplet combination that was greylisted 8 hours ago
> (and never returned) and one that has already returned (which should still
> be whitelisted). This is, of course, unless the whitelist was separate from
> the greylist and whitelist was checked first. I am not sure I want to do
> that. I think I would prefer all entries in one database "table" so the I/O
> is kept down to the minimum.

Use another directory and rename (requires 2 stat checks) or link them
across (requires 2 unlinks to clean up) to whitelist them.  

> Of course, I might just be totally ignorant of how to use the filesystem as
> a database with empty files.

They don't have to be empty, but if they aren't then minimum allocation
rules start to apply.  For example, each 1 byte file in most filesystems
on Linux will use 4kB.  Of course, disk is cheap, so this probably isn't
worth worrying about in the long run.  However, even 2 stat calls should
be faster than the open+read+close needed to read file contents.

> Does any think that the queue algorithm in qmail would be of any use?

I don't see how.  It depends on what part of the "queue algorithm"
you're referring to.

> Or how
> about tai coding expiry time into filename? Shudder... would mean a grep to
> find a match of triplet perhaps.

No, having to list a directory to find one entry would not be efficient
or scalable.

> It seems from my reading that a smarter system is possible with a record
> oriented database. Then separate fields can record first access, last
> access, hit/block/fail counts, expiry times, and whitelist flags.

Yes, if you want to record all this, you will need to either store some
data in the files or move to a more advanced DBM.

> This extra
> info would allow things like extended expiry times for ips that meet certain
> criteria (sort of automatic whitelisting), and selection of a grouping
> function to handle server farms not sending from the same ip, and delaying
> of 451 error to cope with brain dead sender verification scheme. These to
> name a few. With one I/O.

With one I/O you can do the same in a plain text file, and get the bonus
of being able to manipulate the database with command-line tools.

If text files don't cut it, going to a *SQL system would likely be the
next best bet, as the command-line interfaces work well for arbitrary
manipulations of the data.  SQL systems typically give many more
advanced features that simply aren't available when using plain files.

There are still many cases for which even the most efficient *SQL
manager will not be able to match the speed and ease of use of using the
filesystem directly as a database.  For all SQL systems (except SQLite),
you will have to connect, authenticate, select a database, issue a
query, the server parses the query, reads the data, and sends it back.
Compare that to open, read, close (plus a simple parse), and you see why
I have a preference for files.

>  Is there more info than the mtime of a file that could be used
> and manipulated?

Much more.  You could use some of the mode bits as a bitmask, create
extra links to adjust the link count, stuff it with empty data to adjust
the size.  You can also set the access time, but any subsequent open or
read will also set this.
-- 
Bruce Guenter <[email protected]>                http://untroubled.org/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFf4nO6W+y3GmZgOgRAmy/AJkBg8LKs9nwjFiu8UUs4sGY9AoGyACfeUH1
Wc8Ei3i0dD5VGmwYARo22lg=
=MeZS
-----END PGP SIGNATURE-----