Re: Excess memory usage growth for a process

Hi Folks,

So I have been running this Red Gate Memory Profiler for more than 8 =
days
now and I can see some very interesting developments. The Profiler is
monitoring some of the Performance Counters. One in particular has =
caught my
interest and that is the size of Private Bytes. Although steady for the
first two days at 48 MB, after that it started to grow slowly and what =
was a
flat line before, turned into a ascending incline ramp. At the moment I =
type
this message, the size of the Private Bytes is 855 MB. In the task =
manager,
my process is indicated as having the Virtual Memory size up to 876 MB =
while
the Mem Usage is only 64 MB.
I believe this clearly indicates that there is no memory leak per se, =
but
rather is just that the large objects heap is growing w/o control.=20
Is my assessment correct?

Regards,
Eddie


-----Original Message-----
From: Discussion of advanced .NET topics.
[mailto:[email protected]] On Behalf Of Trey Nash
Sent: Thursday, July 30, 2009 2:24 PM
To: [email protected]
Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a process

Hi,

Yes, ADPlus is really just a complex VB script that drive windb(a)g.  If =
you
feed it the -? option, it should spew out a lot of helpful info on the
various ways you can use it.

I wrote a blog post on our team blog not too long ago regarding ADPlus =
that
may be helpful to become more familiar with it:

http://blogs.msdn.com/ntdebugging/archive/2009/05/18/capturing-adplus-clr=
-cr
ashes.aspx

I hope this helps,

	-Trey

> -----Original Message-----
> From: Discussion of advanced .NET topics. [mailto:ADVANCED-
> [email protected]] On Behalf Of Eddie Lascu
> Sent: Thursday, July 30, 2009 1:04 PM
> To: [email protected]
> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a =
process
>=20
> Thank you, Guys. I will look into that as well. Do I need a special
> start of
> the process if I am to use ADPlus or I can attach to the process =
later?
>=20
>=20
> -----Original Message-----
> From: Discussion of advanced .NET topics.
> [mailto:[email protected]] On Behalf Of Greg Young
> Sent: Thursday, July 30, 2009 1:52 PM
> To: [email protected]
> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a =
process
>=20
> Yep exactly what ADPlus is meant for! you can even do this right
> within ADPlus :)
>=20
> On Thu, Jul 30, 2009 at 1:49 PM, Wilson,
> Phil<[email protected]>
> wrote:
> > Not answering this particular question, but I believe you could hook
> all
> this together to automatically get that dump when the memory gets =
large
> if
> it's rare enough that you don't want to sit and look at it all day.
> Perfmon
> has triggers that can fire off a script, and you can fire off the
> debugging
> script ADPlus to dump your process with the -hang option.
> >
> > Phil Wilson
> >
> >
> > -----Original Message-----
> > From: Discussion of advanced .NET topics.
> [mailto:[email protected]] On Behalf Of Eddie Lascu
> > Sent: Wednesday, July 29, 2009 11:40 AM
> > To: [email protected]
> > Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
> process
> >
> > Based on my preliminary reading of what this has to offer, it seems
> you
> can
> > integrate it with your VS IDE (not an option for me as my processes
> are
> > already deployed) or use it via WinDbg, but you need to look at the
> debug
> > version or the runtime version that has the .pdb file (not an option
> either
> > as I only have the sole runtime file deployed, but much easily
> fixable).
> Is
> > this statement accurate?
> >
> > Thanks,
> > Eddie
> >
> > -----Original Message-----
> > From: Discussion of advanced .NET topics.
> > [mailto:[email protected]] On Behalf Of Adam
> Tuliper
> > Sent: Wednesday, July 29, 2009 1:24 PM
> > To: [email protected]
> > Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
> process
> >
> >
> http://blogs.msdn.com/alejacma/archive/2009/06/30/sos-cheat-sheet-net-
> 2-0-3-
> > 0-3-5.aspx
> >
> > !DumpHeap -min 85000
> > !EEHeap -GC; !DumpHeap <LOH begin> <LOH allocated>
> > =A0 =A0-> Shows all objects in Large Object Heap (LOH/Gen 3).
> >
> >
> >
> >
> > On Wed, Jul 29, 2009 at 1:06 PM, Greg Young =
<[email protected]>
> wrote:
> >
> >> Yes with ants you have to wait for the problem to happen.
> >>
> >> if you google SOS you can find some tutorials on it (it is a
> separate
> >> download from windbg ... it can also be used in visual studio!).
> John
> >> Robbins also has a great book covering the subject that I highly
> >> recommend.
> >>
> >> Cheers,
> >>
> >> greg
> >>
> >> On Wed, Jul 29, 2009 at 1:01 PM, Eddie Lascu<[email protected]>
> wrote:
> >> > Shoot, I didn't know SOS can do that for you. Can you share more
> about
> >> this
> >> > tool (a link to download it)? Is it part of the WinDbg tool that
> comes
> >> from
> >> > Microsoft? It's true that with ANTS I am at the mercy of the
> system,
> >> hoping
> >> > it will go nuts again and will have the memory size shut up crazy
> high.
> >> >
> >> > Cheers,
> >> > Eddie
> >> >
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Discussion of advanced .NET topics.
> >> > [mailto:[email protected]] On Behalf Of Greg
> Young
> >> > Sent: Wednesday, July 29, 2009 12:46 PM
> >> > To: [email protected]
> >> > Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
> process
> >> >
> >> > The wonderful thing about SOS is you dont need to run a profiler
> ...
> >> > when it happens take a memory dump, look whats actually happening
> in
> >> > the running app, is your LOH fragmented? Otherwise you are just
> >> > sticking your thumb up your butt and saying it will rain next
> >> > thursday.
> >> >
> >> > On Wed, Jul 29, 2009 at 12:43 PM, Eddie =
Lascu<[email protected]>
> >> wrote:
> >> >> I am using Reg Gate ANTS to profile the app, but it may take too
> much
> > to
> >> >> happen again. Before the spike I saw last weekend, the processes
> ran
> > for
> >> > 3-4
> >> >> months with no significant change in the size of memory used.
> >> >> Will look into Process.WorkingSet.
> >> >>
> >> >> Thanks,
> >> >> Eddie
> >> >>
> >> >>
> >> >>
> >> >> -----Original Message-----
> >> >> From: Discussion of advanced .NET topics.
> >> >> [mailto:[email protected]] On Behalf Of Greg
> Young
> >> >> Sent: Wednesday, July 29, 2009 12:38 PM
> >> >> To: [email protected]
> >> >> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
> process
> >> >>
> >> >> Task manager shows you working set not total memory used but yes
> the
> >> >> process class can help you with that (Process.WorkingSet as an
> >> >> example).
> >> >>
> >> >> Have you verified the LOH being the problem by using SOS?
> >> >>
> >> >> Cheers,
> >> >>
> >> >> Greg
> >> >>
> >> >> On Wed, Jul 29, 2009 at 12:33 PM, Eddie
> Lascu<[email protected]>
> >> wrote:
> >> >>> Hi guys,
> >> >>>
> >> >>> In light of this article:
> >> >>>
> >> >>>
> >> >>
> >> >
> >>
> >
> http://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-
> large-ob
> >> >> ject-heap/
> >> >>>
> >> >>> I am no longer sure what I have is a memory leak. I could see
> how
> some
> >> of
> >> >> the things that happen in my application match the scenarios
> described
> >> in
> >> >> the above article. Hence, I am ready to think about a =
workaround.
> My
> >> >> services are set up to run automatically and restart in case =
they
> > crash.
> >> >> While they never crash, I could leverage that feature and use it
> to
> fix
> >> > this
> >> >> excess memory usage that I experience every once in a blue moon.
> What
> I
> >> >> thought was to monitor the memory used and terminate the
> application
> in
> >> > case
> >> >> it exceeds a set threshold. The Windows Service Manager will =
then
> >> restart
> >> >> the service and all the memory will be reclaimed by the OS.
> >> >>>
> >> >>> My question at this point is whether the .NET Framework has =
some
> > metric
> >> >> classes that can help monitor the total amount of memory
> allocated
> >> within
> >> > a
> >> >> process. Surely System.Diagnostics must have something like =
that.
> Is
> >> > anyone
> >> >> aware of a way to find out what memory is used by a process, =
just
> like
> >> the
> >> >> Task Manager does?
> >> >>>
> >> >>> Cheers,
> >> >>> Eddie
> >> >>>
> >> >>>
> >> >>> -----Original Message-----
> >> >>> From: Discussion of advanced .NET topics.
> >> >> [mailto:[email protected]] On Behalf Of Shawn
> >> > Wildermuth
> >> >>> Sent: Monday, July 27, 2009 12:49 PM
> >> >>> To: [email protected]
> >> >>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
> > process
> >> >>>
> >> >>> Also, your 'using' statement stuff should help, but are you =
sure
> that
> >> all
> >> >>> your object (XML classes and db connection classes) that =
support
> >> >> IDisposable
> >> >>> are being cleaned up?
> >> >>>
> >> >>> Thanks,
> >> >>>
> >> >>> Shawn Wildermuth
> >> >>> http://wildermuth.com
> >> >>> https://agilitrain.com
> >> >>> Microsoft MVP (C#), MCSD.NET, Author and Speaker
> >> >>>
> >> >>> The Silverlight Tour is coming to a city near you!
> >> >>>
> >> >>>
> >> >>> -----Original Message-----
> >> >>> From: Discussion of advanced .NET topics.
> >> >>> [mailto:[email protected]] On Behalf Of John
> Warner
> >> >>> Sent: Monday, July 27, 2009 12:28 PM
> >> >>> To: [email protected]
> >> >>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
> > process
> >> >>>
> >> >>> I don't disagree with you just note that before 'some change'
> working
> >> OK
> >> >>> no memory leaks, after change, memory leak. We both know your
> code
> did
> >> > not
> >> >>> recompile itself to include a memory leak ...
> >> >>> That reminds has the framework/runtimes been updated/patched?
> >> >>>
> >> >>> John Warner
> >> >>>
> >> >>>
> >> >>>> -----Original Message-----
> >> >>>> From: Discussion of advanced .NET topics. [mailto:ADVANCED-
> >> >>>> [email protected]] On Behalf Of Eddie Lascu
> >> >>>> Sent: Monday, July 27, 2009 12:00 PM
> >> >>>> To: [email protected]
> >> >>>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for =
a
> >> >>>> process
> >> >>>>
> >> >>>> Hi John,
> >> >>>>
> >> >>>> I could check that with our system administrators. Do you =
think
> that
> >> can
> >> >>> be
> >> >>>> a possible explanation?
> >> >>>> I just talked with one of the DB Admin and she said she made
> some
> >> >>> changes
> >> >>>> on
> >> >>>> some Stored Produces, but nothing as dramatic as a patch
> update. She
> >> >>> also
> >> >>>> said that the drivers for the network cards were updated last
> week,
> >> but
> >> >>> I
> >> >>>> can't see how that can affect the memory used by my processes.
> >> >>>>
> >> >>>> Thanks,
> >> >>>> Eddie
> >> >>>>
> >> >>>>
> >> >>>> -----Original Message-----
> >> >>>> From: Discussion of advanced .NET topics.
> >> >>>> [mailto:[email protected]] On Behalf Of
> >> >>>> John Warner
> >> >>>> Sent: Monday, July 27, 2009 11:49 AM
> >> >>>> To: [email protected]
> >> >>>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for =
a
> >> >>>> process
> >> >>>>
> >> >>>> Not what you are seeking, but has anything on the server
> changed? An
> >> >>>> Oracle patch or something of that nature?
> >> >>>>
> >> >>>> John Warner
> >> >>>>
> >> >>>>
> >> >>>> > -----Original Message-----
> >> >>>> > From: Discussion of advanced .NET topics. [mailto:ADVANCED-
> >> >>>> > [email protected]] On Behalf Of Eddie Lascu
> >> >>>> > Sent: Monday, July 27, 2009 11:33 AM
> >> >>>> > To: [email protected]
> >> >>>> > Subject: [ADVANCED-DOTNET] Excess memory usage growth for a
> >> >>>> process
> >> >>>> >
> >> >>>> > Hi guys,
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > I need your help with an issue I am having right now. I have
> > several
> >> >>>> windows
> >> >>>> > services deployed. These services gather some XML files from
> the
> >> >>>> internet
> >> >>>> > and then parse the content and save the information in the
> > database.
> >> >>> The
> >> >>>> > database is Oracle 11g, so the access is done through
> ODP.NET. I
> >> had
> >> >>>> some
> >> >>>> > issues before (and some of you with good memory may recall =
my
> >> previous
> >> >>>> > questions) where I noticed the memory used by these services
> > growing
> >> >>>> very
> >> >>>> > high. At the time, I understood that some classes in the
> ODP.NET
> >> >>> library
> >> >>>> use
> >> >>>> > some JAVA code and they are known for not cleaning properly
> after
> >> >>> them.
> >> >>>> As a
> >> >>>> > result I rewrote my code in such a way that every dynamic
> > allocation
> >> >>> of
> >> >>>> an
> >> >>>> > object from the ODP.NET library was encompassed in an =
"using"
> >> >>> statement,
> >> >>>> > like this:
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > using (OracleCommand objDbCommand =3D new
> >> >>>> > OracleCommand(strSqlSelectStatement,
> >> >>>> > objConnection))
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > All was fine for a very long time. All my services kept a
> constant
> >> >>> size
> >> >>>> when
> >> >>>> > it came to the total memory used, a size that hovered around
> 55-60
> >> MB.
> >> >>>> This
> >> >>>> > was the case for months. At the beginning I checked the size
> almost
> >> on
> >> >>> a
> >> >>>> > daily basis. After a while, my confidence grew and I only
> checked
> > it
> >> >>>> once a
> >> >>>> > week or so. Every time I looked at the size of all my
> processes (I
> >> had
> >> >>> 4
> >> >>>> > that followed pretty much the same approach on dealing with
> those
> >> >>> Oracle
> >> >>>> > objects) the size was around that 55-60 MB mark.
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > Now, all of a sudden, this last weekend, the size for some =
of
> the
> >> >>>> services
> >> >>>> > shut up to about 690 MB, despite the fact the service was
> running
> >> with
> >> >>>> the
> >> >>>> > exact same code as it did a month ago. This represents a
> potential
> >> >>> fatal
> >> >>>> > problem for the server that can get to a point where it can
> no
> >> longer
> >> >>>> > allocate memory, so I need to find a way to understand what
> the
> > heck
> >> >>> is
> >> >>>> > going on. Unfortunately, I do not have any metrics
> implemented in
> > my
> >> >>>> > application to log the total amount of memory used. That
> would
> have
> >> >>> been
> >> >>>> a
> >> >>>> > very good indication on when exactly did this surge started
> to
> >> happen.
> >> >>> I
> >> >>>> > could have checked the Event Viewer to see if the server got
> hit
> by
> >> a
> >> >>>> > nuclear bomb or something on that precise moment. I really
> don't
> > get
> >> >>>> this. I
> >> >>>> > mean that's why we moved to .NET in the first place, to be
> rid of
> >> such
> >> >>>> > concerns, like making sure all dynamically allocated objects
> are
> >> then
> >> >>>> > destroyed.
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > Can anyone give me an idea on how to investigate this issue?
> Are
> >> there
> >> >>>> tools
> >> >>>> > out there that could be used to pick at the memory allocated
> by a
> >> >>>> process
> >> >>>> > and see what are those objects that take up so much =
=A0memory?
> For
> > the
> >> >>>> time
> >> >>>> > being, I have the processes running, but two of them are at
> or
> > close
> >> >>> to
> >> >>>> 690
> >> >>>> > MB and I don't know how long will I be able to keep them
> running.
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > If you had an issue like this, what would you try to do?
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > Any suggestion will be greatly appreciated.
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > TIA,
> >> >>>> >
> >> >>>> > Eddie
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> >>>> > View archives and manage your subscription(s) at
> >> >>>> > http://peach.ease.lsoft.com/archives
> >> >>>>
> >> >>>> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> >>>> View archives and manage your subscription(s) at
> >> >>>> http://peach.ease.lsoft.com/archives
> >> >>>>
> >> >>>> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> >>>> View archives and manage your subscription(s) at
> >> >>>> http://peach.ease.lsoft.com/archives
> >> >>>
> >> >>> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> >>> View archives and manage your subscription(s) at
> >> >>> http://peach.ease.lsoft.com/archives
> >> >>>
> >> >>> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> >>> View archives and manage your subscription(s) at
> >> >> http://peach.ease.lsoft.com/archives
> >> >>>
> >> >>> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> >>> View archives and manage your subscription(s) at
> >> >> http://peach.ease.lsoft.com/archives
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Les erreurs de grammaire et de syntaxe ont =E9t=E9 incluses pour
> m'assurer
> >> >> de votre attention
> >> >>
> >> >> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> >> View archives and manage your subscription(s) at
> >> >> http://peach.ease.lsoft.com/archives
> >> >>
> >> >> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> >> View archives and manage your subscription(s) at
> >> > http://peach.ease.lsoft.com/archives
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Les erreurs de grammaire et de syntaxe ont =E9t=E9 incluses pour
> m'assurer
> >> > de votre attention
> >> >
> >> > =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> > View archives and manage your subscription(s) at
> >> > http://peach.ease.lsoft.com/archives
> >> >
> >> > =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> > View archives and manage your subscription(s) at
> >> http://peach.ease.lsoft.com/archives
> >> >
> >>
> >>
> >>
> >> --
> >> Les erreurs de grammaire et de syntaxe ont =E9t=E9 incluses pour
> m'assurer
> >> de votre attention
> >>
> >> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> View archives and manage your subscription(s) at
> >> http://peach.ease.lsoft.com/archives
> >>
> >
> > =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > View archives and manage your subscription(s) at
> > http://peach.ease.lsoft.com/archives
> >
> > =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > View archives and manage your subscription(s) at
> http://peach.ease.lsoft.com/archives
> >
> > =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > View archives and manage your subscription(s) at
> http://peach.ease.lsoft.com/archives
> >
>=20
>=20
>=20
> --
> Les erreurs de grammaire et de syntaxe ont =E9t=E9 incluses pour =
m'assurer
> de votre attention
>=20
> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> View archives and manage your subscription(s) at
> http://peach.ease.lsoft.com/archives
>=20
> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> View archives and manage your subscription(s) at
> http://peach.ease.lsoft.com/archives

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
View archives and manage your subscription(s) at
http://peach.ease.lsoft.com/archives

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
View archives and manage your subscription(s) at http://peach.ease.lsoft.com/archives