Re: Excess memory usage growth for a process

Now that you've got the memory creep happening, take a crash dump and
see what's on the various heaps:

Tess gives the best rundown
http://blogs.msdn.com/tess/archive/2008/02/15/net-debugging-demos-lab-3-mem=
ory.aspx

Also useful:

http://msdn.microsoft.com/en-us/magazine/cc163528.aspx
http://www.eggheadcafe.com/articles/20060114.asp
http://www.dumpanalysis.org/blog/index.php/2007/08/19/crash-dump-analysis-p=
atterns-part-20b/

Things are tons easier with SOSEX v2

http://www.stevestechspot.com/SOSEXV2NowAvailable.aspx


On Thu, Aug 6, 2009 at 14:35, Eddie Lascu<[email protected]> wrote:
> Hi Folks,
>
> So I have been running this Red Gate Memory Profiler for more than 8 days
> now and I can see some very interesting developments. The Profiler is
> monitoring some of the Performance Counters. One in particular has caught=
 my
> interest and that is the size of Private Bytes. Although steady for the
> first two days at 48 MB, after that it started to grow slowly and what wa=
s a
> flat line before, turned into a ascending incline ramp. At the moment I t=
ype
> this message, the size of the Private Bytes is 855 MB. In the task manage=
r,
> my process is indicated as having the Virtual Memory size up to 876 MB wh=
ile
> the Mem Usage is only 64 MB.
> I believe this clearly indicates that there is no memory leak per se, but
> rather is just that the large objects heap is growing w/o control.
> Is my assessment correct?
>
> Regards,
> Eddie
>
>
> -----Original Message-----
> From: Discussion of advanced .NET topics.
> [mailto:[email protected]] On Behalf Of Trey Nash
> Sent: Thursday, July 30, 2009 2:24 PM
> To: [email protected]
> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a process
>
> Hi,
>
> Yes, ADPlus is really just a complex VB script that drive windb(a)g. =C2=
=A0If you
> feed it the -? option, it should spew out a lot of helpful info on the
> various ways you can use it.
>
> I wrote a blog post on our team blog not too long ago regarding ADPlus th=
at
> may be helpful to become more familiar with it:
>
> http://blogs.msdn.com/ntdebugging/archive/2009/05/18/capturing-adplus-clr=
-cr
> ashes.aspx
>
> I hope this helps,
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0-Trey
>
>> -----Original Message-----
>> From: Discussion of advanced .NET topics. [mailto:ADVANCED-
>> [email protected]] On Behalf Of Eddie Lascu
>> Sent: Thursday, July 30, 2009 1:04 PM
>> To: [email protected]
>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a process
>>
>> Thank you, Guys. I will look into that as well. Do I need a special
>> start of
>> the process if I am to use ADPlus or I can attach to the process later?
>>
>>
>> -----Original Message-----
>> From: Discussion of advanced .NET topics.
>> [mailto:[email protected]] On Behalf Of Greg Young
>> Sent: Thursday, July 30, 2009 1:52 PM
>> To: [email protected]
>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a process
>>
>> Yep exactly what ADPlus is meant for! you can even do this right
>> within ADPlus :)
>>
>> On Thu, Jul 30, 2009 at 1:49 PM, Wilson,
>> Phil<[email protected]>
>> wrote:
>> > Not answering this particular question, but I believe you could hook
>> all
>> this together to automatically get that dump when the memory gets large
>> if
>> it's rare enough that you don't want to sit and look at it all day.
>> Perfmon
>> has triggers that can fire off a script, and you can fire off the
>> debugging
>> script ADPlus to dump your process with the -hang option.
>> >
>> > Phil Wilson
>> >
>> >
>> > -----Original Message-----
>> > From: Discussion of advanced .NET topics.
>> [mailto:[email protected]] On Behalf Of Eddie Lascu
>> > Sent: Wednesday, July 29, 2009 11:40 AM
>> > To: [email protected]
>> > Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> process
>> >
>> > Based on my preliminary reading of what this has to offer, it seems
>> you
>> can
>> > integrate it with your VS IDE (not an option for me as my processes
>> are
>> > already deployed) or use it via WinDbg, but you need to look at the
>> debug
>> > version or the runtime version that has the .pdb file (not an option
>> either
>> > as I only have the sole runtime file deployed, but much easily
>> fixable).
>> Is
>> > this statement accurate?
>> >
>> > Thanks,
>> > Eddie
>> >
>> > -----Original Message-----
>> > From: Discussion of advanced .NET topics.
>> > [mailto:[email protected]] On Behalf Of Adam
>> Tuliper
>> > Sent: Wednesday, July 29, 2009 1:24 PM
>> > To: [email protected]
>> > Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> process
>> >
>> >
>> http://blogs.msdn.com/alejacma/archive/2009/06/30/sos-cheat-sheet-net-
>> 2-0-3-
>> > 0-3-5.aspx
>> >
>> > !DumpHeap -min 85000
>> > !EEHeap -GC; !DumpHeap <LOH begin> <LOH allocated>
>> > =C2=A0 =C2=A0-> Shows all objects in Large Object Heap (LOH/Gen 3).
>> >
>> >
>> >
>> >
>> > On Wed, Jul 29, 2009 at 1:06 PM, Greg Young <[email protected]>
>> wrote:
>> >
>> >> Yes with ants you have to wait for the problem to happen.
>> >>
>> >> if you google SOS you can find some tutorials on it (it is a
>> separate
>> >> download from windbg ... it can also be used in visual studio!).
>> John
>> >> Robbins also has a great book covering the subject that I highly
>> >> recommend.
>> >>
>> >> Cheers,
>> >>
>> >> greg
>> >>
>> >> On Wed, Jul 29, 2009 at 1:01 PM, Eddie Lascu<[email protected]>
>> wrote:
>> >> > Shoot, I didn't know SOS can do that for you. Can you share more
>> about
>> >> this
>> >> > tool (a link to download it)? Is it part of the WinDbg tool that
>> comes
>> >> from
>> >> > Microsoft? It's true that with ANTS I am at the mercy of the
>> system,
>> >> hoping
>> >> > it will go nuts again and will have the memory size shut up crazy
>> high.
>> >> >
>> >> > Cheers,
>> >> > Eddie
>> >> >
>> >> >
>> >> >
>> >> > -----Original Message-----
>> >> > From: Discussion of advanced .NET topics.
>> >> > [mailto:[email protected]] On Behalf Of Greg
>> Young
>> >> > Sent: Wednesday, July 29, 2009 12:46 PM
>> >> > To: [email protected]
>> >> > Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> process
>> >> >
>> >> > The wonderful thing about SOS is you dont need to run a profiler
>> ...
>> >> > when it happens take a memory dump, look whats actually happening
>> in
>> >> > the running app, is your LOH fragmented? Otherwise you are just
>> >> > sticking your thumb up your butt and saying it will rain next
>> >> > thursday.
>> >> >
>> >> > On Wed, Jul 29, 2009 at 12:43 PM, Eddie Lascu<[email protected]>
>> >> wrote:
>> >> >> I am using Reg Gate ANTS to profile the app, but it may take too
>> much
>> > to
>> >> >> happen again. Before the spike I saw last weekend, the processes
>> ran
>> > for
>> >> > 3-4
>> >> >> months with no significant change in the size of memory used.
>> >> >> Will look into Process.WorkingSet.
>> >> >>
>> >> >> Thanks,
>> >> >> Eddie
>> >> >>
>> >> >>
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Discussion of advanced .NET topics.
>> >> >> [mailto:[email protected]] On Behalf Of Greg
>> Young
>> >> >> Sent: Wednesday, July 29, 2009 12:38 PM
>> >> >> To: [email protected]
>> >> >> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> process
>> >> >>
>> >> >> Task manager shows you working set not total memory used but yes
>> the
>> >> >> process class can help you with that (Process.WorkingSet as an
>> >> >> example).
>> >> >>
>> >> >> Have you verified the LOH being the problem by using SOS?
>> >> >>
>> >> >> Cheers,
>> >> >>
>> >> >> Greg
>> >> >>
>> >> >> On Wed, Jul 29, 2009 at 12:33 PM, Eddie
>> Lascu<[email protected]>
>> >> wrote:
>> >> >>> Hi guys,
>> >> >>>
>> >> >>> In light of this article:
>> >> >>>
>> >> >>>
>> >> >>
>> >> >
>> >>
>> >
>> http://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-
>> large-ob
>> >> >> ject-heap/
>> >> >>>
>> >> >>> I am no longer sure what I have is a memory leak. I could see
>> how
>> some
>> >> of
>> >> >> the things that happen in my application match the scenarios
>> described
>> >> in
>> >> >> the above article. Hence, I am ready to think about a workaround.
>> My
>> >> >> services are set up to run automatically and restart in case they
>> > crash.
>> >> >> While they never crash, I could leverage that feature and use it
>> to
>> fix
>> >> > this
>> >> >> excess memory usage that I experience every once in a blue moon.
>> What
>> I
>> >> >> thought was to monitor the memory used and terminate the
>> application
>> in
>> >> > case
>> >> >> it exceeds a set threshold. The Windows Service Manager will then
>> >> restart
>> >> >> the service and all the memory will be reclaimed by the OS.
>> >> >>>
>> >> >>> My question at this point is whether the .NET Framework has some
>> > metric
>> >> >> classes that can help monitor the total amount of memory
>> allocated
>> >> within
>> >> > a
>> >> >> process. Surely System.Diagnostics must have something like that.
>> Is
>> >> > anyone
>> >> >> aware of a way to find out what memory is used by a process, just
>> like
>> >> the
>> >> >> Task Manager does?
>> >> >>>
>> >> >>> Cheers,
>> >> >>> Eddie
>> >> >>>
>> >> >>>
>> >> >>> -----Original Message-----
>> >> >>> From: Discussion of advanced .NET topics.
>> >> >> [mailto:[email protected]] On Behalf Of Shawn
>> >> > Wildermuth
>> >> >>> Sent: Monday, July 27, 2009 12:49 PM
>> >> >>> To: [email protected]
>> >> >>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> > process
>> >> >>>
>> >> >>> Also, your 'using' statement stuff should help, but are you sure
>> that
>> >> all
>> >> >>> your object (XML classes and db connection classes) that support
>> >> >> IDisposable
>> >> >>> are being cleaned up?
>> >> >>>
>> >> >>> Thanks,
>> >> >>>
>> >> >>> Shawn Wildermuth
>> >> >>> http://wildermuth.com
>> >> >>> https://agilitrain.com
>> >> >>> Microsoft MVP (C#), MCSD.NET, Author and Speaker
>> >> >>>
>> >> >>> The Silverlight Tour is coming to a city near you!
>> >> >>>
>> >> >>>
>> >> >>> -----Original Message-----
>> >> >>> From: Discussion of advanced .NET topics.
>> >> >>> [mailto:[email protected]] On Behalf Of John
>> Warner
>> >> >>> Sent: Monday, July 27, 2009 12:28 PM
>> >> >>> To: [email protected]
>> >> >>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> > process
>> >> >>>
>> >> >>> I don't disagree with you just note that before 'some change'
>> working
>> >> OK
>> >> >>> no memory leaks, after change, memory leak. We both know your
>> code
>> did
>> >> > not
>> >> >>> recompile itself to include a memory leak ...
>> >> >>> That reminds has the framework/runtimes been updated/patched?
>> >> >>>
>> >> >>> John Warner
>> >> >>>
>> >> >>>
>> >> >>>> -----Original Message-----
>> >> >>>> From: Discussion of advanced .NET topics. [mailto:ADVANCED-
>> >> >>>> [email protected]] On Behalf Of Eddie Lascu
>> >> >>>> Sent: Monday, July 27, 2009 12:00 PM
>> >> >>>> To: [email protected]
>> >> >>>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> >> >>>> process
>> >> >>>>
>> >> >>>> Hi John,
>> >> >>>>
>> >> >>>> I could check that with our system administrators. Do you think
>> that
>> >> can
>> >> >>> be
>> >> >>>> a possible explanation?
>> >> >>>> I just talked with one of the DB Admin and she said she made
>> some
>> >> >>> changes
>> >> >>>> on
>> >> >>>> some Stored Produces, but nothing as dramatic as a patch
>> update. She
>> >> >>> also
>> >> >>>> said that the drivers for the network cards were updated last
>> week,
>> >> but
>> >> >>> I
>> >> >>>> can't see how that can affect the memory used by my processes.
>> >> >>>>
>> >> >>>> Thanks,
>> >> >>>> Eddie
>> >> >>>>
>> >> >>>>
>> >> >>>> -----Original Message-----
>> >> >>>> From: Discussion of advanced .NET topics.
>> >> >>>> [mailto:[email protected]] On Behalf Of
>> >> >>>> John Warner
>> >> >>>> Sent: Monday, July 27, 2009 11:49 AM
>> >> >>>> To: [email protected]
>> >> >>>> Subject: Re: [ADVANCED-DOTNET] Excess memory usage growth for a
>> >> >>>> process
>> >> >>>>
>> >> >>>> Not what you are seeking, but has anything on the server
>> changed? An
>> >> >>>> Oracle patch or something of that nature?
>> >> >>>>
>> >> >>>> John Warner
>> >> >>>>
>> >> >>>>
>> >> >>>> > -----Original Message-----
>> >> >>>> > From: Discussion of advanced .NET topics. [mailto:ADVANCED-
>> >> >>>> > [email protected]] On Behalf Of Eddie Lascu
>> >> >>>> > Sent: Monday, July 27, 2009 11:33 AM
>> >> >>>> > To: [email protected]
>> >> >>>> > Subject: [ADVANCED-DOTNET] Excess memory usage growth for a
>> >> >>>> process
>> >> >>>> >
>> >> >>>> > Hi guys,
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > I need your help with an issue I am having right now. I have
>> > several
>> >> >>>> windows
>> >> >>>> > services deployed. These services gather some XML files from
>> the
>> >> >>>> internet
>> >> >>>> > and then parse the content and save the information in the
>> > database.
>> >> >>> The
>> >> >>>> > database is Oracle 11g, so the access is done through
>> ODP.NET. I
>> >> had
>> >> >>>> some
>> >> >>>> > issues before (and some of you with good memory may recall my
>> >> previous
>> >> >>>> > questions) where I noticed the memory used by these services
>> > growing
>> >> >>>> very
>> >> >>>> > high. At the time, I understood that some classes in the
>> ODP.NET
>> >> >>> library
>> >> >>>> use
>> >> >>>> > some JAVA code and they are known for not cleaning properly
>> after
>> >> >>> them.
>> >> >>>> As a
>> >> >>>> > result I rewrote my code in such a way that every dynamic
>> > allocation
>> >> >>> of
>> >> >>>> an
>> >> >>>> > object from the ODP.NET library was encompassed in an "using"
>> >> >>> statement,
>> >> >>>> > like this:
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > using (OracleCommand objDbCommand =3D new
>> >> >>>> > OracleCommand(strSqlSelectStatement,
>> >> >>>> > objConnection))
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > All was fine for a very long time. All my services kept a
>> constant
>> >> >>> size
>> >> >>>> when
>> >> >>>> > it came to the total memory used, a size that hovered around
>> 55-60
>> >> MB.
>> >> >>>> This
>> >> >>>> > was the case for months. At the beginning I checked the size
>> almost
>> >> on
>> >> >>> a
>> >> >>>> > daily basis. After a while, my confidence grew and I only
>> checked
>> > it
>> >> >>>> once a
>> >> >>>> > week or so. Every time I looked at the size of all my
>> processes (I
>> >> had
>> >> >>> 4
>> >> >>>> > that followed pretty much the same approach on dealing with
>> those
>> >> >>> Oracle
>> >> >>>> > objects) the size was around that 55-60 MB mark.
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > Now, all of a sudden, this last weekend, the size for some of
>> the
>> >> >>>> services
>> >> >>>> > shut up to about 690 MB, despite the fact the service was
>> running
>> >> with
>> >> >>>> the
>> >> >>>> > exact same code as it did a month ago. This represents a
>> potential
>> >> >>> fatal
>> >> >>>> > problem for the server that can get to a point where it can
>> no
>> >> longer
>> >> >>>> > allocate memory, so I need to find a way to understand what
>> the
>> > heck
>> >> >>> is
>> >> >>>> > going on. Unfortunately, I do not have any metrics
>> implemented in
>> > my
>> >> >>>> > application to log the total amount of memory used. That
>> would
>> have
>> >> >>> been
>> >> >>>> a
>> >> >>>> > very good indication on when exactly did this surge started
>> to
>> >> happen.
>> >> >>> I
>> >> >>>> > could have checked the Event Viewer to see if the server got
>> hit
>> by
>> >> a
>> >> >>>> > nuclear bomb or something on that precise moment. I really
>> don't
>> > get
>> >> >>>> this. I
>> >> >>>> > mean that's why we moved to .NET in the first place, to be
>> rid of
>> >> such
>> >> >>>> > concerns, like making sure all dynamically allocated objects
>> are
>> >> then
>> >> >>>> > destroyed.
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > Can anyone give me an idea on how to investigate this issue?
>> Are
>> >> there
>> >> >>>> tools
>> >> >>>> > out there that could be used to pick at the memory allocated
>> by a
>> >> >>>> process
>> >> >>>> > and see what are those objects that take up so much =C2=A0memo=
ry?
>> For
>> > the
>> >> >>>> time
>> >> >>>> > being, I have the processes running, but two of them are at
>> or
>> > close
>> >> >>> to
>> >> >>>> 690
>> >> >>>> > MB and I don't know how long will I be able to keep them
>> running.
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > If you had an issue like this, what would you try to do?
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > Any suggestion will be greatly appreciated.
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > TIA,
>> >> >>>> >
>> >> >>>> > Eddie
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> >>>> > View archives and manage your subscription(s) at
>> >> >>>> > http://peach.ease.lsoft.com/archives
>> >> >>>>
>> >> >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> >>>> View archives and manage your subscription(s) at
>> >> >>>> http://peach.ease.lsoft.com/archives
>> >> >>>>
>> >> >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> >>>> View archives and manage your subscription(s) at
>> >> >>>> http://peach.ease.lsoft.com/archives
>> >> >>>
>> >> >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> >>> View archives and manage your subscription(s) at
>> >> >>> http://peach.ease.lsoft.com/archives
>> >> >>>
>> >> >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> >>> View archives and manage your subscription(s) at
>> >> >> http://peach.ease.lsoft.com/archives
>> >> >>>
>> >> >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> >>> View archives and manage your subscription(s) at
>> >> >> http://peach.ease.lsoft.com/archives
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Les erreurs de grammaire et de syntaxe ont =C3=A9t=C3=A9 incluses =
pour
>> m'assurer
>> >> >> de votre attention
>> >> >>
>> >> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> >> View archives and manage your subscription(s) at
>> >> >> http://peach.ease.lsoft.com/archives
>> >> >>
>> >> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> >> View archives and manage your subscription(s) at
>> >> > http://peach.ease.lsoft.com/archives
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Les erreurs de grammaire et de syntaxe ont =C3=A9t=C3=A9 incluses p=
our
>> m'assurer
>> >> > de votre attention
>> >> >
>> >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> > View archives and manage your subscription(s) at
>> >> > http://peach.ease.lsoft.com/archives
>> >> >
>> >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> > View archives and manage your subscription(s) at
>> >> http://peach.ease.lsoft.com/archives
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Les erreurs de grammaire et de syntaxe ont =C3=A9t=C3=A9 incluses pou=
r
>> m'assurer
>> >> de votre attention
>> >>
>> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> >> View archives and manage your subscription(s) at
>> >> http://peach.ease.lsoft.com/archives
>> >>
>> >
>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> > View archives and manage your subscription(s) at
>> > http://peach.ease.lsoft.com/archives
>> >
>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> > View archives and manage your subscription(s) at
>> http://peach.ease.lsoft.com/archives
>> >
>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> > View archives and manage your subscription(s) at
>> http://peach.ease.lsoft.com/archives
>> >
>>
>>
>>
>> --
>> Les erreurs de grammaire et de syntaxe ont =C3=A9t=C3=A9 incluses pour m=
'assurer
>> de votre attention
>>
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> View archives and manage your subscription(s) at
>> http://peach.ease.lsoft.com/archives
>>
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> View archives and manage your subscription(s) at
>> http://peach.ease.lsoft.com/archives
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> View archives and manage your subscription(s) at
> http://peach.ease.lsoft.com/archives
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> View archives and manage your subscription(s) at http://peach.ease.lsoft.=
com/archives
>



--=20
Marc C. Brooks
http://musingmarc.blogspot.com

NOT sent from an iPhone

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
View archives and manage your subscription(s) at http://peach.ease.lsoft.com/archives