Re: what journal options should I use on linux?

Merge tables aren't an easy option here. There's a higher level
data partitioning that we're putting into place, though, and it's
been shown to help a lot. But I also want to get as much out of the
file system as I can.

> The "solution" I use for duplicate filtering is to to create a
> "load" table which loads a set of records, then compares those
> records against the merge table for duplicates, deleting any found.
> Then the load table is added to the merge table and the process is
> repeated for the next batch of data.

I don't think this will help us, but it's an interesting technique.
We use staging tables to cut the load in a bunch of places. 

I think the true answer to this particular problem lies outside SQL
and instead with a private index structure that is tuned for dealing
with duplicates...it would help if the MyISAM engine was a little
more clever about really large indexes.

Best,
chris

-- 
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe:    http://lists.mysql.com/[email protected]