Re: GZIP

By the way, there's a simple metadata mechanism already in place based on
filename suffixes. You can register different serializers for journals and
snapshots based on suffix. For example, I've used this mechanism for
versioning and data migration: *.v1snapshot, *.v2snapshot, etc. It could
also be used to indicate a compressed snapshot, e.g. *.gzsnapshot. For
snapshots, there is always one current version (all snapshots _written_
with the same serializer) but other versions can still be read, with
migration code built into the serializer. For journals, it's more for
validation -- I always discourage trying to migrate journals (because it's
hard to maintain identical behavior), so Prevayler refuses to open journals
with anything other than the current suffix.


On Mon, Jan 23, 2012 at 10:55 AM, Karl Wettin <[email protected]> wrote:

> I suppose another but rather hacky solution would be to gzip manually and
> then open it up using a gzipstream if it match some criteria such as file
> extension. They all have the same magic numbers...
>
> Perhaps the criteria could come from some sort of registry containing meta
> data such as serializer class and compression for journals and snapshots?
> It could be as simple as 0000000000000000001.journal.meta.xml
> and 0000000000000000001.snapshot.meta.xml with a small utility class that
> would post process files at snapshot time, the previous journals on
> startup, or what not. I suppose this also would make it simpler for people
> to switch serializer half way in to the game, if that ever was a problem.
>
>
>
>
> karl
>
>
> On Jan 23, 2012, at 7:19 PM, Justin T. Sampson wrote:
>
> It makes conceptual sense, but it turns out to be tricky. The journal
> needs to ensure that data is synced to disk at specific moments, and
> compression and encryption tend to have their own internal structure that
> complicates that. Perhaps the ideal would be invoking compression and
> encryption at the moment when DurableOutputStream writes a batch to disk,
> which would indeed require inserting a hook in the middle of the
> DurableOutputStream code (since it can't simply be a wrapper above or below
> that level).
>
> On Wed, Jan 18, 2012 at 11:52 AM, Klaus Wuestefeld <[email protected]> wrote:
>
>> Maybe a hook, like the serializer.
>>
>>
>> On Tue, Jan 17, 2012 at 4:27 PM, Karl Wettin <[email protected]>
>> wrote:
>> > I just managed to compress a GZIPSerializer built journal from 70MB to
>>  5MB using gzip the.journal.
>> >
>> > GZIPSerializer deflates each instance with a new header. This means
>> that snapshots are compressed fully while journals is far from it.
>> >
>> > I'm starting to think that compression is something that should be
>> implemented in DurableOutputStream.
>> >
>> > What do you say?
>> >
>> >
>> >
>> >                        karl
>> >
>>
>

------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2

_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org