Re: ideas for returning memory to the OS

* Boris Zbarsky <[email protected]> wrote:

Hi,

> > That's one of the major reasons why mozilla is so fat
> 
> You mean in terms of codebase size?

Yes, and also memory consumption and required development/maintenance
resources.

> Part of the problem, you see, is that OS libraries have a tendency to be broken. 

Right. Then try to fix these libraries (within their jobs) or tell
the world about the broken crap.

If you try to work around evrybody else's crap, you'll never get finished.
And once one of the bugs is fixed, you have to take care of disabling or
removing the workaround (since it will most likely break then, again)

> In different ways on different OSes.  

The trick in cross-platform development is not caring about every single
difference, but abstraction: build upon interfaces which are DEFINED to
be platform independent (and maybe help implementing these within their
dedicated projects).

In other words: strict modularity.

> Or sometimes not broken but simply not behaving in the desired way 
> (locale-insensitive floating point to string conversions comes to mind) 

If you need new semantics: build a new, dedicated library and *import*
(NOT include) it into your main project.

> Which means that writing cross-platform code involves either building 
> in lots of exceptions for this brokenness or reimplementing 
> parts of libc (see NSPR).

Platform dependend exceptions (workarounds) are the worst approach:
For an large application, it makes it unmaintainable, and all that
work is very redundant.

Reimplementing some functionality (eg. of libc) is unproblematic, 
as long as it's done in an dedicated library, which doesn't conflict
existing ones and is open to the world.

> > and why socialist (top-down) economy only works on
> > very small scale.
> 
> Actually, the reason for this is quite different.  It has more to 
> do with so the model only working if there is broad agreement about 
> the desired course of action and trust amongst all participants.  

No, this is *EXACTLY* the reason. Your argumentation has the problem
that it implies these (intra-company) difference are somehow invalid.
In fact they may be very valid, but the equilibrium phantasies of 
higher management layers are the fault.

Different purposes (products, services, business targets) require
also different approaches, different ways of thinking and working.
Once you have one entity in charge for all of them, the risk of
failing is very hight.

For example, several years ago, and german big store company (Hertie)
tried an massive expansion by buying and trying to "integrate" lots
of other - very different - companies, eg. large estate (whole shopping
arcades), asset management, mail order shops (Neckermann, Quelle) up 
to gastronomy (Starbuck's) and . Obviously they failed with an big crash. 
The top-level management simply tried something unduable.

The socialist (top-down planned) economy of the GDR had similar 
complexity problems: it was completely directed on very few ultimate
goals which didn't (and never ever could) reflect the needs of the 
people. There was no room for individuality and so innovation stalled
(for a few year's we've been better than capitalist nations in some 
areas, but just only for a few years). So, in the last 15 to 20 years
we had an massive economic degeneration.

In history, there have been similar economic models, which managed to
be working quite stable for long time, but these were just quite small
societies. For example, an south american city with a up to few million
people, which had been destroyed in an massive millitary driven genozide
by european invadors.

The scalability of top-down models is very limited. 

Several large companies have learned these lessons and split themselves
off into lots of smaller subs - leaving the main one just as holding.
This gives them more flexibility, eg. to sell an sub and/or replace
it by someone else.

> > The mozilla project itself should not take any influence to other
> > projects like glibc. Of course, mozilla folks are welcomed to join
> > these other projects, but then functioning as developers of these
> > projects instead of moz there.
> 
> You didn't address the key point.  The typical situation is that a bug 
> needs fixing. 

For a *short* time, it's okay to work around other's bugs. But this has
to be *strictly limited*, or you'll end up in unmaintainable complexity
sooner or later.

> It can be fixed in libc, or it can be fixed on the caller end by not 
> using libc. The next libc update that will include the fix it it gets 
> fixed in libc today will ship in 18 months.

The long release cycles of certain packages (eg. glibc) have their 
reasons: the same we're currently talking about. Simply too large!

IMHO it's better to help solving the actual problem instead of trying
to work around it (and so introduce the risk of new bugs).

> I think everyone accepts that the best possible scenario is that the 
> OS handles the things it's supposed to handle. The problem is that in 
> reality sometimes it doesn't. 

Then it might be broken. The OS vendor is responsible for providing 
an working OS - no one else.

> One could even argue that in some areas (memory allocation, scheduler) 
> it can't always behave optimally because different workloads require 
> such wildly different behavior and because it doesn't have all the 
> information needed to behave optimally (e.g. information about future 
> lifetimes of allocations that the application may have).

That's an scenario I already mentioned above: 
If you need new semantics, you also need a new interface. Obviously.
So, in this example, we would implement some new memory allocator
which provides our new interface.

I don't see any reason why that new allocator should be limited to
the only one client package. IMHO this is a big waste of resources
(development time as well as user's machine resources)
 
> > Or in business slang: "concentrate on your key competences".
> 
> Which means paying someone else to do the things you're not very 
> good at. However, if you can't find such a someone else at a price 
> you can afford, you might end up doing them yourself, even if the 
> result is not as good as it could have been.

Right. But in most cases you'll have to invest into proper resources.
In longer terms you can get much more out of them if you offer your
newly learned/built capabilities to others. An wise controller will
threat that new resources as its own company, and often they're 
actually moved out to separated ones. (That's the way many service
and supplier companies are spinned-off larger companies).

> > On a qick look through the current seamonkey tree, I've seen
> > at lot of bundled 3rd-party packages things which really should
> > NOT belong there:
> 
> I'll let the people more familiar with some of these speak about 
> why they're in the tree.  

Some time ago, I already asked for the reasons. For those few cases
I got an answer, it were just people not wanting to wait for some
bugs get fixed.

Right, sometimes upstream can be too slow in fixing certain bugs
(especially if they dont believe they're important, eg. dozens of 
broken makefiles which break clean crosscompiling and packaging ;-o)
That's the reason why many distros maintain their own patchsets and
its exactly what the OSS-QM project (http://oss-qm.metux.de/) is for.

> For cairo in particular it has to do with the fact that we don't 
> necessarily want to be building against system cairo (which is 
> often outdated right now).

Ah, so you also feel responsible for certain people's broken our 
outdated systems ? Oh, that's interesting.

Maybe you also feel responsible for my dirty dishes and like to 
come around for cleaning the up ? ;-P
 
BTW: choosing certain versions of imported libs is cleanly an 
build-time (or even deploy-time) issue. As long as you do your 
imports in an clean standard way, aka. using pkg-config (and 
calling it via the $PKG_CONFIG variable), each packager/sysop
can easily maintain it. I don't see why the moz devel should 
waste resources on that.

> >> I'm sure Robert's
> >> comment was more about what to do on shorter timescale like for
> >> Gecko 1.9 / FF 3.
> > 
> > Do you have an pointer to it ?
> 
> To Gecko 1.9?

To the comments you're refering to.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------