Migration hurdles (and some possible coping strategies)

In "[Zope2-migration] New mailing list" message, Paul Everitt hits a 
large number of migration nails directly candidly on their heads:

Paul is revealing some very important catch-22's and gotchas which make 
the migration difficult and/or risky.  I'll elaborate where I think I 
have something to expand on or some suggestions.  Overall, I agree 
wholeheartedly with the message and issues, including

	Like you, I'll caveat all this with a disclaimer: things aren't this
	gloomy, this is a hard problem and I also don't have silver bullets,
	etc.  However, this doesn't excuse us from the responsibility of
	managing this transition and the risks involved.  There's more to the
	transition than writing new code letting the chips fall where they may.

So, with that important caveat in mind, I'll expand where I can and 
offer a few possible insights.  I do not have any silver bullets but 
just some ideas.

 >Because they want "Zope in general" to succeed?

I think the crucial issue to most outsiders is survival of their 
career, job, and project.  Zope2 is enough to keep people's hands 
full... learning, upgrading, grokking.  Zope3 is periphery (or worse 
threatening) until it's easy to use, test, migrate to.  IOW, I very 
much agree with Paul when he candidly theorizes:

	the maturity level of Zope 2 + CMF.  Some people won't change just for
	a new CA, if it doesn't do something they deem important.

I believe that's absolutely true and will go further.  Zope's best and 
smartest (not necessarily brightest in the sense of leading edge, but 
smartest in the sense of responsibly managing risks for their projects) 
will absolutely *not* migrate until the benefits clearly outweigh the 
risks and downsides.  Next-generation Zope needs to overcome those 
risks, or reduce the barriers to entry as thoroughly as possible.

 >However, Zope 2
 >is a mature project that is beginning to appeal to people in the
 >mainstream.  I think a poor transition could have a negative effect in
 >this part of Zopeland.

Agreed.  That leads to perhaps the most annoying (but natural and 
inevitable) issue.  The issue of migrating to Zope2004(Z3) can have a 
negative impact on the uptake of Zope2.  So dealing with the problem 
successfully in advance will help both Zope2 maintain its growth 
pattern and ultimately Zope2004(Z3) achieve its potential.

FYI, I think Apple did a pretty good job migrating from OS 9 to Mac OS 
X.  On OS 9, they provided a forward compatibility box (libraries) to 
ease migration off OS 9 toward OS X while still running OS 9.  In fact 
some applications deploy to either OS 9 or OS X using the compatibility 
box.  Then, on OS X, they provided a backward compatibility box to ease 
the transition onto OS X from OS 9 once running OS X but before fully 
adopting its new API's, etc..  Even then, some (e.g., the high provile 
Quark) struggled, but the bulk migrated smoothly even though the two 
OS's are radically different.

A similar forward push (forward compatibility box from Z2 to Z2004/Z3) 
and backward pull (backward compatibility to Z2 from Z2004/Z3) would 
provide two major ways to ease migration from old to new.

 >   "List the top 5 factors in Zope that help you sell it.  List the top
 >5 that make it hard."
 >
 >The challenge, though, is what happens if we find out:
 >
 >   Top 5 Needs of Deployers != Top 5 Interests of CA Developers

(Stupid question: What is "CA"?  Content...Application?)

I do not have any silver bullets, but I do have two (err, make that: 
three) ideas (feel free to take with grain(s) of salt as desired).  
Briefly,

1.  Provide an explicit staged approach to the transition.  This 
entails a coarse grained list of features as they relate to Zope2004/Z3 
releases.  The first Z2004 release would have to provide baseline 
functionality, e.g., folders, url mapping to html pages, etc..  The 
next release may include TAL (I'm making this up but that's the idea), 
incrementally increasing functionality with each successive minor 
release.  Each Z2004 release could incorporate new Z3 features as well 
as some backward compatible Z2 features.  So, TAL support could be 
accompanied by DTML support (grafted on).  The stages would 
(presumably) add simpler features early on, and more complex, esoteric 
features toward the end.
	With a project map of stages, prospective upgraders could look at the 
map and see where they fit into it.  Less advanced users would get a 
working Z2004 earlier since their features are likely supported earlier 
on.  Advanced features would support advanced users later on, but with 
the project map, anyone could experiment with certain known-to-work 
features in the earlier part of the lifecycle and get a taste for 
what's coming.
	Each stage would have a date, Z2 compatibility milestones, Z2004/Z3 
technology milestones, and a stability rating (since that's important 
as well).  The project map provides a taste of what's coming, when, and 
where it's likely worth experimenting with.  The map also serves as a 
vehicle to check progress with the outside world, and get their 
feedback on how the compatibility strategy is working.  (Perhaps 
volunteer effort could help tune the backwards compatibility efforts?)  
Each functional release could be accompanied by a request for 
compatibility testing and conversion testing by community.  With each 
release, this would be an early invitation to get feet wet with 
Z2004/Z3 by testing the familiar (Z2) in the context of the new 
(Z2004/Z3).  With luck, this process would provide a succession of 
slippery slopes until the community was adding considerable value to 
the conversion process and, possibly, informing the final Z2004/Z3 
design as well.
	I think this projectmap could help provide a bird's eye view for those 
of us who are not attuned to Zope's advanced development.  This roadmap 
idea will be visited more visually in a follow-on post.

2.  Offer features in Z2004/Z3 which are so desirable that the upgrade 
is not just comforting, evolutionary, but a genuine 
no-brainer/wish-list sort of home-run (foreshadow of grain of salt goes 
here).  See below for radical example (appended rather than core).  
Such a radical upgrade could also generate new interest in Zope on its 
own, depending on its nature.

3.  [On the fly addition as corollary to 2, above.]  Beg, borrow, cheat 
and steal from other projects and open standards to a. adopt familiar 
preexisting (open standard) solutions, and b. save work by not having 
to reinvent preexisting wheels.  IOW, offer less special stuff and more 
open standards stuff in Z2004/Z3 than in Z2.  That's a nice carrot 
everyone can understand and which would actually provide Z2004/Z3 as a 
reduction in risk compared to Z2.  (Downside is less lock-in to Zope, 
but somehow I do not believe the Zope steering committee is purposely 
aspiring to proprietary lock-in; upsides include less perceived risk, 
less maintenance (since outside teams maintain open stuff), and a 
higher profile for Zope since it becomes a prominent 
contributor/maintainer to best-of-breed open source technology.  Apple 
seems to be succeeding on this front as well with adoption and support 
of BSD, X11 (intro'd last month), Rendezvous (discovery networking), 
etc..)  Open standards prevailing over Zope proprietary standards could 
balance the risks of adopting a "new" Zope.
	One great value in this open standards strategy is also that scarce 
resources (developer time) can be spent on what really matters: adding 
unique Zope value rather than reinventing fairly standard wheels.  I do 
not know how this relates specifically to Z2, Z2004/3 feature lists, 
but I'm hoping someone with a bird's eye view may find value in this 
approach.

That's about it, except for my radical aside example of strategy #2 
which I've appended, below.

= Joe =

Example of Feature offering Radical No-Brainer Appeal:

[Yes, I know Zope-on-Java(Jython) has been rejected in the past by Zope 
inner circle as Java has been cast as a competitor to Zope, though I 
disagree with that notion utterly.  Nevertheless, that's the current 
situation as I understand it.  I know others feel the way I do about 
Zope-on-Java, as I know of two attempts to migrate Zope to Java; 
unfortunately, both projects died prematurely.
	To me, Zope-on-Java is so obvious, it seems almost inevitable.  So, 
that's why I've chosen to present it.  However, due to the 
controversial and radical nature of the proposal, I've chosen to keep 
it as an aside. Presumably, some other ideas may exist involving XML or 
other web technologies which could satisfy this strategy (to some 
degree).  Others are better equipped to fill in the blanks as 
appropriate.  However, my chosen no-brainer upgrade would be the 
Zope-on-Java, so that's what I'll go with here as an example.]

Even given the Java as competitor hurdle above, there are additional 
hurdles to this strategy (as I see it): Python and Jython upgrade 
dependencies.  These could probably be staged out to minimize or avoid 
them, but the ideal (and simplest) version involves them, so I'll go 
with that.

-=-

Caveats to the brute-force, idealistic approach of Zope-on-Java:

1.  Python would need to upgrade to Python3000 (?) where numbers are 
not necessarily objects.  This version would need to remain JVM (Java 
VM) compatible so that it could be cleanly, efficiently implemented on 
the JVM, e.g., integers would be implmented as java integers, and longs 
would need special handling (hopefully still reasonably efficient, just 
not pure jvm type efficient).  This provides a huge speed increase to 
Jython, and presumably Zope/Jython would be much faster than 
Zope/CPython (whereas now Zope is CPython "slow", and Zope/Jython (were 
it to exist) would be even slower due to low-level C code --> Java).
	Bad news/good news:
	Bad news:  I see no reference to Python3000 on the python.org site or 
FAQ, so I'm not really clear what happened to the next generation 
python project or whether it even exists anymore.
	Good news:  Zope3 is rumored to reduce/minimize C code dependencies 
compared to Zope2, thereby easing the migration of Zope onto Java 
(i.e., less C code to recreate in Java code; more pure python).  This 
notion was even expressed by a ZopeCorp insider.  I hope that means 
(fingers crossed) the Zope us/them attitude toward Java may finally be 
shifting; however, I'm still chalking that up to wishful thinking until 
I hear ZopeCorp express a java friendly view explicitly.

2.  Jython needs support.  Python is on 2.3, while Jython is only on 
2.1.  Jython is amazing, but not heavily supported (yet?).  For clean 
support of a pure python Z3, Jython needs to be current with Python.

Pragmatically, these issues can be sidestepped with enough elbow grease 
and perseverance (and maintenance).  Python3000 could be circumvented 
by creating a Zope-on-Java specific jython which takes the efficiency 
path to new levels (mostly for speedups in jython implementation).  
However, this approach is a bit unsavory due to branching to 
non-standard implementation (or playing maverick); it's probably not 
that bad but would require extra effort and maintenance.  
Unfortunately, it's probably necessary even with the Python3000 
approach since P3000 would have its own (slow) schedule, so anything 
useful would be a require using a prototype implementation before 
becoming Python standard.

For Jython issues, the pragmatic issue would be to backport Zope to 
Python2.2 (assuming that's most up-to-date Jython support at the time). 
  That would work but requires extra effort and presumably has a few 
hurdles as well if certain Python2.3 features are key to Z3.

-=-

Ok, given those caveats, where's the win?  First of all, Java is a huge 
beast with an inordinate number of powerful libraries and open source 
technologies, not to mention attractive corporate deployments and 
dollars.  Zope could adopt some of java technologies even to the 
replacement of some of its own (DTML comes to mind).  This would 
involve Zope depending on and contributing back to other open source 
projects (a la Apple with BSD, X11, and Rendezvous technologies).  That 
would be yet another shift in the ZopeCorp/Zope business model.
	Zope technologies may even become more widely used in this scenario 
(by exporting Zope technology in java).  In any case, Zope would 
definitely pick up more mind share from its contributions and 
participation on other open source projects, and with luck could ride 
new technologies supported by a wider (than strictly Zope) community.

Also, I know that ZopeCorp (really, board of directors, I believe) is 
at least sensitive to the notion that Java is a comfort zone for many 
enterprises, whereas Zope is not (nor really python to much degree).  
Zope-on-Java provides a valuable perceptual signature that Zope is 
based on a current, standard, and familiar platform (since it uses 
Java).  I agree it's perception, but that's market reality.

In this case, however, it's nice that market reality dovetails with 
lots of powerful code, e.g., JSP compatibility would be real nice for 
people to try Zope but keep their existing JSP sites up and running.  
With Java compatibility, Zope could run pure (as desired) or in hybrid 
mode where it communicates nicely with existing Java based technologies 
from preexisting sites.  IOW, java compatibility would provide both 
backward (JSP) and forward (future java technologies) compatibility for 
Zope.  I suspect this would only be a good thing for growth to the Zope 
community.

Wow, there's a long aside!

Back to point:  If you want Z2004/Z3 to gain the upper hand in the 
risk/reward tradeoff calculation for existing (and prospective) zope 
users, provide a no-brainer upgrade feature which is virtually 
irresistable.  Zope-on-Java is the most compelling one with which I am 
familiar, however, there could well be some rabbits up your sleeves 
which you could dovetail onto Z2004/Z3 to provide such no-brainer 
irresistability.  I would suggest brainstorming for such hidden gold 
(XML tech?) and enticements as an incentive to migrate.