Re: Re: stackless implementation (was Re: New stacklessproposal)

Mark Hahn wrote:

> What you have done is written OS threads that are harder to implement than
> real OS threads.  You have all the problems of saving and restoring the
> registers as usual plus the extra problems of reusing the same stack with
> the copying.  You will probably be slower than an OS switch but you will
> have the advantage of using less memory.

Quite right, but:
My channel implementation with hard-switching as above
is 5 times faster than OS switching, and with soft-switching
it is over 10 times faster.

It was always true: Real threads give you almost all you can
do with tasklets. What makes the difference is that I would
never do this, because it is too slow.
Certain algorithms begin to become interesting only if
their speed compares well to other techniques.

> So we have three types of threads (mixed is stackless 2.0):
> 
> Pure OS:  High memory.  Medium speed switch.  C extensions allowed.  No
> pickling.
> 
> Mixed:  Low memory.  Lower speed switch.  C extensions allowed.  No
> Pickling.
> 
> Pure interpreter: Low memory.  High speed switch.  No C extensions.
> Pickling OK.

Actually there are four, and I have to adjust the ranking:

Pure OS:      High memory. Low speed. C extensions OK. No pickling.

Mixed:        Medium memory. Medium speed. C extensions OK. No pickling.

Pure interp.: Low memory. High speed. No C extensions. Pickling OK.

Stack pooled: Low memory. Higher speed. Either C extensions or pickling.

> I was originally thinking we would only have the pure OS and the pure
> interpreter.  Now we have the new more complex mixed type.  There are no new
> language features required for the mixed.  The mixed just removes some
> severe restrictions.
> 
> Should we give the programmer some control over which of the three they get
> other than the obvious that they cannot use a c extension with the pure
> interpreter?

Well, for testing purposes, I have a global switch which tells the
interpreter whether to try soft-switching or blindly swap stacks
all the time. Usually the programmer wants it fast.
The nesting problem can be detected, automatically.

> It is a shame this is getting so complicated.

You bet. I'm working on Stackless since five years now.
The Stackless 2.0 startover took just two weeks until
I had something running. The combined model is not trivial.

> It would be nice to make it
> automatic so we could hide the complexity from the programmer.  I wonder if
> any algorithm could be smart enough to always make the right choice for
> them?

Yes, I think so, in most circumstances.

> The only time the programmer is really screwed is if they try to use c
> extensions and pickling.  Every other combination is supported in the
> matrix.

By using a C extension, they prevend pickling. But also only
in this context. If they can defer the pickling until the
callback is over, pickling is fine.

...

> In your last message you were talking of having multiple real C stacks.  I
> assume you were not going to let the OS do the C stack switching?  You are
> starting to replace OS functions.

No, why should I leave to the OS what I can do much better.
I already have everything for the switching.
The code for stack hi-jacking can be easily re-used for pure
switching. What the system does is *way* too much. The nice
thing with application-level stack switching is that you know
what your interpreter uses, saving a lot of work.

Furthermore, doing this is completely different from using OS
features. Evn with switched stacks, my approach is still a single
OS thread, and GUIs like wxWindows will be working happily
from any tasklet, concurrently.

ciao - chris
-- 
Christian Tismer             :^)   <mailto:[email protected]>
Mission Impossible 5oftware  :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34  home +49 30 802 86 56  mobile +49 173 24 18 776
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/