real-world architectural refactoring in a dirty environment?

Folks;

hoping not to be all too off-topic with this here, I am looking for
some fundamental insight in how others do architectural refactoring on
a semi-large scale. Here's the situation:

* Most of our business logic lives as customizing code inside a
  document management system. The amount of code in there has
  accumulated throughout the last 8 .. 10 years, most of the time being
  implemented in "quick hacks" to tickle a very specific customer need.

* There is no substantial data model or architecture. Most of the
  customizing code is rather tightly coupled to / woven into the
  infrastructural aspects dictated by the document management system.

* The system itself is built on top of a proprietary programming
  language which just exists inside this very system. There is no
  development, monitoring, testing, deployment, administration tooling
  support at hand. 

* The platform, as is, has no notion of an "external API" or support
  for _any_ kind of interface technology you might want to use. There
  is very crippled CORBA support (unusable to solve real-world
  problems), there is very crippled SOAP support, and there is next to
  no support for even things as fundamental as writing schema-validable
  XML. The only way of making this thing appear inside of a more
  "up-to-date" environment is a hand-crafted (and thoroughly buggy,
  thanks to the limitations of the language usable for that)
  HTTP-GET/POST structure transferring something akin to "JSON lite". 

At the moment, modifying, extending, maintaining the code base becomes
more and more difficult, as complexity has by far reached a degree at
which it has become next to unfeasible. We suffer from seemingly
non-deterministic performance problems at times. We know that
introducing new features takes ages simply due to struggling and
fighting with the platform woes. We have quite to some extent dealt
with refactoring and "brownfield" approaches, yet always and forever
again ending up with the problem that, in this environment, we can't
simply pull functionality out (or even refactor the thing as a whole
initially) because we can't ensure that, after messing with the code,
things still work, because we can't do reliable tests.

So, this is the mess. We're on the edge between trying to pursue a
slow, step-by-step migration away from this (to a more up-to-date
environment to actually allow for meaningful work again) and throwing
it all away and starting over anew from scratch, trying to make these
things better by now. I am not really sure which of these options to
choose, even technically. So, I'd kindly like to ask for any input,
insights, ... into this. Does anyone around here have any experience of
doing refactoring under conditions akin to these outlined above? What
kind of lessons did you learn down that road? Any meaningful way of
really handling with a situation like that?

TIA and all the best,
Kristian


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/refactoring/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/refactoring/join
    (Yahoo! ID required)

<*> To change settings via email:
    [email protected] 
    [email protected]

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/