ODLM/OGFS Recovery

This is a multi-part message in MIME format.

------_=_NextPart_001_01C428B6.3DC29403
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi all,

Attached please find some discussion I'm trying to write to understand
the problems with OpenDLM/OpenGFS recovery after a node dies.  Memexp
had this all integrated, but OpenDLM works quite differently.

Please take a look and comment ... I'm hoping that I'm wrong about some
of the things I wrote, especially about DLM_VALNOTVALID.  Please give me
a sanity check.

I'm thinking about using a list within the lock module to retain locks
that get granted during lock recovery, but must not be forwarded to
OpenGFS yet.

There's more that I need to write, about timing and recovery
notification ... Running the journal recovery after lock recovery is
complete, then notifying all nodes when journal recovery is done.

I'm thinking (just an idea) about using a callback from OpenDLM to the
lock module to indicate:

-- when lock recovery begins (and which node died)
-- when lock recovery ends
-- when journal recovery ends (in response to a call from the lock
module), perhaps using a new BARRIER state in the ODLM recovery state
machine, to wait until all nodes' journal recovery ("client" recovery,
more generically) had occurred.

Also wrestling with multiple filesystems ... Each must recover before
moving on.  Memexp had separate instances and storage areas for each
filesystem, but there's just one ODLM.   Arrgh.

Stan's been thinking about this also, but I haven't captured his
thoughts ... Please add/comment ... And anyone else, too.

-- Ben --

Opinions are mine, not Intel's





------_=_NextPart_001_01C428B6.3DC29403
Content-Type: application/octet-stream;
	name="recovery"
Content-Transfer-Encoding: base64
Content-Description: recovery
Content-Disposition: attachment;
	filename="recovery"

CgoyLiAgT0dGUyBKb3VybmFsIGFuZCBPRExNIExvY2sgUmVjb3ZlcnkKLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCgoyLjEgIE9HRlMgSm91cm5hbCBSZWNvdmVyeQotLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLQoKT0dGUyByZWNvdmVyeSBpbnZvbHZlcyByZXBsYXlpbmcg
dGhlIGpvdXJuYWwgb2YgYSBub2RlIHRoYXQgaGFzIGRpZWQsIHdyaXRpbmcKdG8gZGlzayBhbnkg
bWV0YWRhdGEgdGhhdCBoYWQgYmVlbiBzY2hlZHVsZWQgdG8gYmUgd3JpdHRlbiB0byBkaXNrLCBi
dXQgZmFpbGVkCnRvIGhhcHBlbiBiZWNhdXNlIHRoZSBub2RlIGRpZWQuCgpPR0ZTIHJlcGxheXMg
YSBqb3VybmFsIHdoZW46CgotLSBtb3VudGluZyB0aGUgZmlsZXN5c3RlbQotLSBhbm90aGVyIG5v
ZGUgZGllcyB3aGlsZSB0aGUgZmlsZXlzdGVtIGlzIG1vdW50ZWQKCk5vcm1hbGx5LCB0aGUgcmVw
bGF5IGF0IG1vdW50IHRpbWUgc2hvdWxkICpub3QqIGNhdXNlIGFueSBtZXRhZGF0YSB3cml0ZXMs
CmJlY2F1c2UgdGhlIG1ldGFkYXRhIHNob3VsZCBoYXZlIGJlZW4gZmx1c2hlZCB0byBkaXNrLCBh
bmQgdGhlIGpvdXJuYWwKZW1wdGllZCwgd2hlbiB0aGUgZmlsZXN5c3RlbSB3YXMgcHJldmlvdXNs
eSwgc3VjY2Vzc2Z1bGx5LCB1bm1vdW50ZWQuCgpJdCdzIGxpa2VseSwgdGhvdWdoLCB0aGF0IG5v
ZGUgZGVhdGggd2lsbCByZXN1bHQgaW4gbWV0YWRhdGEgd3JpdGVzIHRvIGRpc2ssCmlmIHRoZSBu
b2RlIHdhcyBpbnZvbHZlZCBpbiB3cml0aW5nIChpbmMuIG1vdmluZyBvciByZW1vdmluZykgaXRl
bXMgb24gdGhlCmRpc2sgYXQgaXRzIHRpbWUgb2YgZGVhdGguCgpJbiBhIG11bHRpLW5vZGUgY2x1
c3RlciwgYW5vdGhlciBub2RlIHdpbGwgcmVwbGF5IHRoZSBqb3VybmFsIG9mIGEgZGVhZCBub2Rl
LApwZXJoYXBzIGZpbmlzaGluZyBiZWZvcmUgdGhlIGRlYWQgbm9kZSByZWJvb3RzIGFuZCByZW1v
dW50cyB0aGUgZmlsZXN5c3RlbS4KSW4gYSBzaW5nbGUtbm9kZSBzeXN0ZW0sIHRoZSBtb3VudCB0
aW1lIHJlcGxheSBzZXJ2ZXMgYXMgdGhlIG5vZGUgZGVhdGggcmVwbGF5LApzaW5jZSB0aGlzIGlz
IHRoZSBmaXJzdCB0aW1lIHRoYXQgdGhlIG9uZSBub2RlIGdldHMgYWNjZXNzIHRvIHRoZSBqb3Vy
bmFsIGFuZApmaWxleXN0ZW0gc2luY2UgdGltZSBvZiBkZWF0aC4KCgoyLjEuMS4gIFNwZWNpYWwg
bG9ja3MgdXNlZCBkdXJpbmcgT0dGUyBNb3VudCBhbmQgUmVjb3ZlcnkKLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCgpPR0ZTIGFzc3VtZXMg
dGhhdCBFWGNsdXNpdmUgKHdyaXRlKSBsb2NrcyBoZWxkIGJ5IGRlYWQgbm9kZXMgd2lsbCBwZXJz
aXN0IGFmdGVyCnRoZSBub2RlJ3MgZGVhdGgsIGFuZCBjb250aW51ZSB0byBwcm90ZWN0IHRoZSBv
YmplY3QgYmVpbmcgd3JpdHRlbiBieSB0aGUgZGVhZApub2RlLiAgV2hpbGUgdGhpcyBpcyB0aGUg
YmVoYXZpb3Igb2YgT0dGUycgbGVnYWN5ICJtZW1leHAiIGxvY2sgcHJvdG9jb2wsIHRoaXMKaXMg
bm90IHRydWUgd2l0aCBPcGVuRExNLCB3aGljaCBjbGVhbnMgKmFsbCogbG9ja3Mgb3duZWQgYnkg
YSBkZWFkIG5vZGUgKGV2ZW4KaWYgbWFya2VkICJPUlBIQU5hYmxlIikuICBUaGUgZGlzY3Vzc2lv
biBiZWxvdyBkZXNjcmliZXMgT0dGUycgZXhwZWN0YXRpb25zOwpvdXIgam9iIHdpbGwgYmUgdG8g
c2ltdWxhdGUgdGhpcyBiZWhhdmlvciB3aXRoaW4gdGhlIG9wZW5kbG0gbG9jayBtb2R1bGUuCgpD
ZXJ0YWluIHNwZWNpYWwgbG9ja3MgYXJlIHJlcXVlc3RlZCBieSBPcGVuR0ZTIHdpdGggdGhlIE5P
RVhQIGZsYWcuICBUaGlzCmZsYWcgbWVhbnMgdGhhdCB0aGUgbG9jayBzaG91bGQgYmUgZ3JhbnRl
ZCB0byAqdGhpcyogbm9kZSBldmVuIGlmIGl0IGlzIGFuCm9ycGhhbiBFWCBsb2NrIGhlbGQgYnkg
YSBkZWFkIG5vZGUuICBUaGlzIGZsYWcgaXMgdXNlZCBvbmx5IGZvciBsb2NrcyB1c2VkIGZvcgpy
ZWNvdmVyeS9tb3VudCBvcGVyYXRpb25zOgoKT3BlbkdGUyBNb3VudCBsb2NrIC0tIEVYLCAqbm90
KiB0aGUgc2FtZSBhcyB0aGUgT3BlbkRMTSBsb2NrIG1vZHVsZSBNb3VudCBsb2NrLgpPcGVuR0ZT
IExpdmUgbG9jayAtLSBhIHNoYXJlZCBsb2NrLCBzbyBhY3R1YWxseSB3b3VsZCBuZXZlciBiZSBh
biBvcnBoYW4KCShhbmQgSSBoYXZlbid0IHNlZW4gZXZpZGVuY2Ugb2YgdGhpcyBwcm90ZWN0aW5n
IGFueXRoaW5nLCBhbnl3YXkpLgpPcGVuR0ZTIFRyYW5zYWN0aW9uIGxvY2sgLS0gdXN1YWxseSBh
IHNoYXJlZCBsb2NrLCBzbyBtdWx0aXBsZSBub2RlcyBjYW4gd3JpdGUKCWRhdGEgdG8gZGlzayBz
aW11bHRhbmVvdXNseSwgYnV0IGlzIGdyYWJiZWQgRVggYnkgT3BlbkdGUwoJb2dmc19yZWNvdmVy
X2pvdXJuYWwoKSwgdG8ga2VlcCBhbnkgb3RoZXIgbm9kZXMgZnJvbSB3cml0aW5nIHRvIGRpc2sK
CXdoaWxlIGEgam91cm5hbCByZWNvdmVyeSBpcyBpbiBwcm9ncmVzcy4gIFRoaXMgd291bGQgYmUg
YW4gb3JwaGFuIGlmIGEKCW5vZGUgZGllZCB3aGlsZSByZXBsYXlpbmcgaXRzIG93biBvciBhbm90
aGVyIG5vZGUncyBqb3VybmFsLgpPcGVuR0ZTIEpvdXJuYWwgbG9ja3MgLS0gRVgsIG9uZSBsb2Nr
IGZvciBlYWNoIGpvdXJuYWwuCgpUaGVzZSBsb2NrcyBtYXkgYmUgaGVsZCBhcyBvcnBoYW5zIGJ5
IGRlYWQgbm9kZShzKS4gIE5PRVhQIG92ZXJyaWRlcyBzdWNoIGhvbGRzLAphbmQgYWxsb3dzIHRo
ZSBsb2NrIHRvIGJlIGdyYWJiZWQgYnkgdGhlIHJlcXVlc3Rpbmcgbm9kZS4gIE5PRVhQIG11c3Qg
Km5vdCoKb3ZlcnJpZGUgbm9uLW9ycGhhbiBsb2NrcyBoZWxkIGJ5IGFueSAobGl2ZSkgbm9kZXMu
ICBUaGF0IHdvdWxkIHZpb2xhdGUgdGhlCm5vcm1hbCBvcGVyYXRpb24gb2YgdGhlIGxvY2tzLgoK
CjIuMiAgT0RMTSBMb2NrIFJlY292ZXJ5Ci0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCgpPRExN
IHJlY292ZXJ5IGludm9sdmVzIGNsZWFuaW5nIG91dCBsb2NrcyBoZWxkIGJ5IGEgbm9kZSB0aGF0
IGRpZWQsIHdoaWxlCnJldGFpbmluZyBhbmQgcmUtZGlzdHJpYnV0aW5nIGFueSB2YWxpZCBpbmZv
cm1hdGlvbiBhYm91dCB0aGUgbG9ja2FibGUgcmVzb3VyY2VzCmFuZCBsb2NrcyBoZWxkIGJ5IHN1
cnZpdmluZyBub2Rlcy4gIE5vdGUgdGhhdCAqbm8qIGxvY2tzIG93bmVkIGJ5IGEgZGVhZCBub2Rl
LApub3QgZXZlbiBvcnBoYW4gbG9ja3MsIHN1cnZpdmUgd2hlbiBhIG5vZGUgZGllcy4gIE9ycGhh
biBsb2NrcyBzdXJ2aXZlIG9ubHkKY2xpZW50IGFwcGxpY2F0aW9uIGRlYXRocywgbm90IG5vZGUg
ZGVhdGhzLgoKUmVjb3ZlcnkgaXMgY29udHJvbGxlZCBieSB0aGUgcmVjb3Zlcnkgc3RhdGUgbWFj
aGluZS4gIFNlZSB0aGUgIm9kbG0tcmVjb3ZlcnkiCmRvYyAoIk9wZW5ETE1fTG9ja19SZWNvdmVy
eSIpIGF0OgoKaHR0cDovL29wZW5kbG0uc291cmNlZm9yZ2UubmV0L2RvY3MucGhwCgpUaGUgcmVj
b3ZlcnkgcHJvY2VzcyBzdGFydHMgc29vbiBhZnRlciByZWNlaXZpbmcgYSBtZXNzYWdlIGZyb20g
dGhlIGNsdXN0ZXIKbWFuYWdlciB1c2VkIGJ5IE9wZW5ETE0sIGluZGljYXRpbmcgdGhhdCBhIG5v
ZGUgaGFzIGRpZWQuICBEdXJpbmcgdGhpcyB0aW1lLApPcGVuRExNIGRvZXMgbm90IHNlcnZpY2Ug
YW55IG5ldyBsb2NrIHJlcXVlc3RzIChkb2VzIGl0IGNvbnRpbnVlIHNlcnZpY2luZwpyZXF1ZXN0
cy1pbi1wcm9ncmVzcz8/KS4gIAoKVGhlIHJlY292ZXJ5IHByb2Nlc3MgZW5kcyBvbmNlIGFsbCBu
b2RlcyBoYXZlIHJlYWNoZWQgdGhlIGZpbmFsICJiYXJyaWVyIgpzdGF0ZSwgYW5kIGhhdmUgbm90
aWZpZWQgZWFjaCBvdGhlciB0aGF0IHRoZXkgY2FuIGFsbCBtb3ZlIGJhY2sgdG8gInJ1biIKbW9k
ZS4gIEFzIHBhcnQgb2YgdGhlIGxhc3Qgc3RhZ2Ugb2YgcmVjb3ZlcnksIHRoZSByZWNvdmVyeSBw
cm9jZXNzIGdyYW50cwpsb2NrcyB0aGF0IHdlcmUgYmxvY2tlZCBieSB0aGUgZGVhZCBub2RlLgoK
VGhlIHJlY292ZXJ5IHByb2Nlc3MgbWF5IHJlc3RhcnQgaWYgYW5vdGhlciBub2RlIGRpZXMsIHNv
IHRoZXJlIGNhbiBiZSBtb3JlCnN0YXJ0cyB0aGFuIGVuZHMuICBUaGUgcmVjb3ZlcnkgcHJvY2Vz
cyBpcyBub3QgcmVjdXJzaXZlLiAgVGhlcmUgaXMgb25seSBvbmUKcmVjb3Zlcnkgc3RhdGUgbWFj
aGluZSwgYW5kIG9ubHkgb25lIHN0YXRlLCBmb3IgYSBnaXZlbiBub2RlLgoKCjIuMyAgQ29vcmRp
bmF0aW5nIEpvdXJuYWwgYW5kIExvY2sgUmVjb3ZlcnkKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLQoKV2UgbXVzdCBwcm90ZWN0IHRoZSBmaWxlcyB0aGF0IHRoZSBk
ZWFkIG5vZGUgd2FzIHdyaXRpbmcsIGJ5IG5vdCBhbGxvd2luZwpvdGhlciBub2RlcyB0byBncmFi
IGxvY2tzIG9uIHRob3NlIGZpbGVzIChlaXRoZXIgcmVhZC9zaGFyZWQgbG9ja3MsIHNpbmNlCnRo
ZXkgbWlnaHQgcmVhZCBiYWQvc3RhbGUgaW5mb3JtYXRpb24sIG9yIHdyaXRlL2V4Y2x1c2l2ZSBs
b2Nrcywgc2luY2UgdGhleQp3b3VsZCBjbG9iYmVyIG9yIGJlIGNsb2JiZXJlZCBieSB0aGUgdXBj
b21pbmcgam91cm5hbCByZXBsYXkpLgoKVGhlIHByb3RlY3Rpb24gbXVzdCBjb250aW51ZSB0aHJv
dWdoIHRoZSBlbmQgb2YgdGhlIGpvdXJuYWwgcmVwbGF5LgpKb3VybmFsIHJlcGxheSBjYW5ub3Qg
c3RhcnQgdW50aWwgT3BlbkRMTSBoYXMgcmVjb3ZlcmVkLCBzaW5jZSBqb3VybmFsIHJlcGxheQpu
ZWVkcyB0byBncmFiIGEgY291cGxlIG9mIGltcG9ydGFudCBsb2NrcyAodHJhbnNhY3Rpb24gYW5k
IGpvdXJuYWwgbG9ja3MpLAp3aGljaCBtdXN0IGJlIGdyYW50ZWQgYnkgT3BlbkRMTSAqYWZ0ZXIq
IGxvY2sgcmVjb3ZlcnkuICBCeSB0aGF0IHRpbWUsCk9wZW5ETE0gd2lsbCBoYXZlICJmb3Jnb3R0
ZW4iIGFsbCBvZiB0aGUgbG9ja3Mgb3duZWQgYnkgdGhlIGRlYWQgbm9kZSwgYW5kCmdyYW50ZWQg
bG9ja3MgdG8gb3RoZXIgd2FpdGluZyBub2Rlcy4KClRoaXMgc3VnZ2VzdHMgdGhhdCB0aGUgbG9j
ayBtb2R1bGUgbXVzdCBpbnRlcnZlbmUsIGFjY2VwdGluZyB0aGUgZ3JhbnRlZApsb2NrcyBmcm9t
IE9wZW5ETE0gZHVyaW5nIGxvY2sgcmVjb3ZlcnksIGJ1dCBrZWVwaW5nIHRoZW0gZnJvbSBPcGVu
R0ZTIAooc2F2aW5nIHRoZW0gb24gYSBsaXN0KSB1bnRpbCBhZnRlciBqb3VybmFsIHJlY292ZXJ5
LiAgVGhpcyAiZmFrZXMiIHRoZSBlZmZlY3QKb2YgYWxsb3dpbmcgdGhlIGRlYWQgbm9kZSdzIGxv
Y2tzIHRvICJwZXJzaXN0Iiwgc2ltcGx5IGJlY2F1c2UgT0dGUyBvbiBvdGhlcgpub2RlcyB3b24n
dCBzZWUgdGhhdCB0aGVpciBsb2NrIHJlcXVlc3RzIGhhdmUgYmVlbiBncmFudGVkLgoKPz9JcyB0
aGVyZSBhIHBvc3NpYmlsaXR5IG9mIGRlYWRsb2NrIGhlcmUgLi4uIGEgbm9kZSBoYXMgYSBzaGFy
ZWQgdHJhbnNhY3Rpb24KbG9jaywgYnV0IGNhbid0IGZpbmlzaCBpdHMgam9iIChhbmQgcmVsZWFz
ZSB0aGUgdHJhbnNhY3Rpb24gbG9jaykgYmVjYXVzZSBpdApjYW4ndCBnZXQsIGUuZy4gYW4gaW5v
ZGUgbG9jaz8/CgoKMi40LiAgU3VwcG9ydGluZyBOT0VYUAotLS0tLS0tLS0tLS0tLS0tLS0tLS0t
CgpUaGUgZGVhZCBub2RlJ3Mgd3JpdGUgbG9ja3Mgd2VyZSBFWCBsb2NrcywgYW5kIHRoZXNlIGFy
ZSB0aGUgb25seSBvbmVzIHRoYXQKbmVlZCB0byAicGVyc2lzdCIsIGF0IGxlYXN0IGluIGVmZmVj
dC4gIEZvcnR1bmF0ZWx5LCBPRExNIGhhcyBhIHdheSBvZgppbmRpY2F0aW5nIHRoYXQgYSBkZWFk
IG5vZGUncyBsb2NrIG1pZ2h0IGhhdmUgYmVlbiBFWCwgdmlhIHRoZSBETE1fVkFMTk9UVkFMSUQK
c3RhdHVzIGJpdCwgd2hpY2ggaW5kaWNhdGVzIHRoYXQgdGhlIExWQiBpcyBub3QgdmFsaWQuICBV
bmZvcnR1bmF0ZWx5LAp0aGlzIGlzIG5vdCBjb25jbHVzaXZlIGluIHRlcm1zIG9mIGluZGljYXRp
bmcgd2hldGhlciwgYWJzb2x1dGVseSwgYSBkZWFkCm5vZGUgaGFkIGhlbGQgYW4gRVggbG9jay4g
IEl0IGNhbiBzaG93IGEgZmFsc2UgcG9zaXRpdmUgb3IgYSBmYWxzZSBuZWdhdGl2ZToKCi0tIGZh
bHNlIHBvc2l0aXZlOiAgT3BlbkRMTSwgZHVyaW5nIGxvY2sgcmVjb3ZlcnksIHdpbGwgbWFyayBh
IHJlc291cmNlJ3MgTFZCCglhcyBpbnZhbGlkIGlmIGl0IGNhbm5vdCBwcm92ZSB0byBpdHNlbGYg
dGhhdCBhIGRlYWQgbm9kZSwgdGhhdCB3YXMgdGhlCglyZXNvdXJjZSBtYXN0ZXIsIGRpZCBub3Qg
b3duIGFuIEVYIGxvY2sgb24gdGhlIHJlc291cmNlLiAgSXQgZG9lcyB0aGlzCglieSBsb29raW5n
IGZvciBFWCwgUFcsIG9yIENXIGxvY2tzIG93bmVkIGJ5IGxpdmUgbm9kZXMsIHdoaWNoIHdvdWxk
CgloYXZlIGtlcHQgdGhlIGRlYWQgbm9kZShzKSBmcm9tIG93bmluZyBhbiBFWCBvciBQVyBhbmQg
d3JpdGluZyB0byB0aGUKCUxWQi4gIElmIGl0IGNhbid0IGZpbmQgYSAibGl2ZSIgRVgvUFcvQ1cg
bG9jaywgaXQgbWFya3MgdGhlIExWQiBpbnZhbGlkLAlqdXN0IHRvIGJlIHNhZmUsIGV2ZW4gdGhv
dWdoIHRoZSBkZWFkIG5vZGUgbWlnaHQgbm90IGhhdmUgYWN0dWFsbHkKCW93bmVkIGFuIEVYL1BX
LiAgU2VlIGNsbXJfbGt2bGIoKS4KCi0tIGZhbHNlIG5lZ2F0aXZlOiAgSWYgdGhlIGRlYWQgbm9k
ZSB3YXMgdGhlIGRpcmVjdG9yeSwgcmVzb3VyY2UgbWFzdGVyLCBhbmQKCW9ubHkgb3duZXIgb2Yg
YSBsb2NrLCBubyBvdGhlciBub2RlIHdvdWxkIGtub3cgYWJvdXQgdGhlIHJlc291cmNlLAoJYW5k
IGFsbCBpbmZvcm1hdGlvbiBhYm91dCB0aGUgbG9jayB3b3VsZCBiZSBsb3N0LiAgU28sIGEgbG9j
ayByZXF1ZXN0Cglmcm9tIG9uZSBvZiB0aGUgc3Vydml2aW5nIG5vZGVzIHdvdWxkIGVuZCB1cCBs
b29raW5nIGxpa2UgYSByZXF1ZXN0IG9uCglhIG5ldyByZXNvdXJjZSAoZG8gdGhlc2UgcmV0dXJu
IERMTV9WQUxOT1RWQUxJRD8pLgoKU28sIHRoZXJlIHNlZW1zIHRvIGJlIHNvbWUgYW1iaWd1aXR5
IGluIGRldGVjdGluZyB3aGV0aGVyIGEgbG9jayBoYWQgYmVlbgpoZWxkIGJ5IGEgZGVhZCBub2Rl
LiAgVGhpcyBpcyB0cm91Ymxlc29tZSB3aGVuIGl0IGNvbWVzIHRvIHRyZWF0aW5nIHRoZQpOT0VY
UCBmbGFnIGFzc2VydGVkIGJ5IE9wZW5HRlMuICBOT0VYUCBzaG91bGQgb3ZlcnJpZGUgb25seSB0
aG9zZSBsb2NrcyB3aGljaApoYWQgYmVlbiBoZWxkIGJ5IGEgZGVhZCBub2RlLCBidXQgbm90IHRo
b3NlIHRoYXQgYXJlIGhlbGQgYnkgbGl2ZSBub2Rlcy4gIEEKZmFsc2UgcG9zaXRpdmUgY291bGQg
Y2F1c2UgYSBwcm9ibGVtIGhlcmUsIG92ZXJyaWRpbmcgYSAoQ1IpIGxvY2sgaGVsZApsZWdpdGlt
YXRlbHkgYnkgYSBsaXZlIG5vZGUsIHN1Y2ggYXMgdGhlIHRyYW5zYWN0aW9uIGxvY2suICBUaGlz
IGNvdWxkIG9jY3VyCmlmIHRoZSBkZWFkIG5vZGUgaGFkIGJlZW4gdGhlIG1hc3RlciBmb3IgdGhl
IHRyYW5zYWN0aW9uIGxvY2suCg==

------_=_NextPart_001_01C428B6.3DC29403--


-------------------------------------------------------
This SF.net email is sponsored by: The Robotic Monkeys at ThinkGeek
For a limited time only, get FREE Ground shipping on all orders of $35
or more. Hurry up and shop folks, this offer expires April 30th!
http://www.thinkgeek.com/freeshipping/?cpg=12297