Re: [AM] [dm-discuss] ZZZ Farewell Fellow Agilists

The biggest problem we have in refactoring databases is not the time it
takes to reload or to test. That is actually trivial (even with very
large databases). Rather it is the coupling between the databases and
the programs that use them. Where I work, we have 30 years of coupling
program code directly to databases, and as a result any change to a
database can affect hundreds of programs. As a result we cannot get
agreement from developers to refactor in small steps. 

Where we can we have abstracted the database away from code using data
modules, which has helped. But even that gets a great deal of pushback
due to having to rework the original programs to use the modules.

I cannot speak for any other data folks, but one of the biggest reasons
we have for a lack of agility in our operational systems (I keep saying
that because we are doing a little better on the DW front, I'm a little
defensive) is the fact that changes to the databases ripple back into
the mass of legacy programs. We're migrating toward a more flexible
situation, but we have a huge amount of inertia to contend with.

Rich

>>> [email protected] 01/31/04 12:06PM >>>
Similarly, a database could in theory be refactored from a
series of small steps, each having well known characteristics,
but in sum having arbitrary complexity.  What I don't know is 
what it would take for this approach to work efficiently.  Are
there things you can do to decouple the database from the
code - accessing through stored procedures only, perhaps?
Are there ways to make automated testing fast enough to
enable small steps?  How can you know for certain that all 
relevant aspects have been tested?  Can we predict
performance, or do we need to find out through experiment?
I don't even know what questions I should be asking here.

> But we have discussed in the DM list some reasons why DBAs
> prefer to batch up requests. E.g. at the most trivial level, add
> several columns to one table. And reasons why database
> refactoring is so costly (e.g. migration of test and live data).

Agreed - the changes for an actual release need to be batched 
up.  It may be that it would be better to make the releases
with greater frequency, but 'in house' releases can act in
place of many releases to the live system - as long as we get
sufficient feedback from the customer w.r.t. the 'in house'
release.

> I think what is needed is strong advice on practical procedures
> and technologies to help DBAs make a change (any change)
> *quicker*. One has to consider
>
> impact analysis
> test data migration
> live data migration
> revision and regression testing of all the code that depends
> on the part of the databse that has changed

One of the reasons we need Data Managers themselves to
try and become agile is that somebody with real in-depth
knowledge needs to look at all these aspects, question
why they are being done, what the goals are, and whether
the processes could be re-arranged to achieve the same
goals in a better manner, one that would support more agility.
What does modern tool support give us that can revloutionise
the way we do the process?  What tool support would be 
needed to achieve this if we aren't ready to go as-is?

I don't see that we can do much about live data migration,
but I would expect something useful could be done about 
all the others.  Have you considered test data generation
instead of test data migration?  How can you give the database
a fixed interface rather than an arbitrary one - and would this 
be too much of a constraint on the business?  A fixed interface
is amenable to regression testing. Where coders can
introduce SQL queries arbitrarily, regression testing will
always be problematic.

If we can run the entire set of regression tests and get them
all passing after a refactoring, does this eliminate the need
for an impact analysis?


> Incidentally, those of us who believe the DBA should be
> responsible for data services don't see this as a data v
> processing thing so much as component refactoring thing.
> It just happens that so many other components depend so
> intimately on the data component. 

I think the dependency is inevitable, but there should be
ways of decoupling and reducing the intimacy of
dependence.  What you have now is the equivalent of
the 'ripple effect' because external users depend not
only on the data but also on how it is structured.
Correct me if I'm wrong here.

> IOW The data services could be SQL or Java, it makes no
> difference at all to the refactoring issues.

IMHO it makes a difference.  SQL is going direct to the data,
and could be impacted by any restructuring.  Java is going
through some interface, and we may be able to stop the
ripple effect from going beyond that interface.


Paul Oldfield

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
www.aptprocess.com

any opinions expressed herein are not necessarily those of
Mentors of Cally or the Appropriate Process Movement
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

For more information about AM, visit the Agile Modeling Home Page at
www.agilemodeling.com

For more information about AM, visit the Agile Modeling Home Page at www.agilemodeling.com
--^----------------------------------------------------------------
This email was sent to: [email protected]

EASY UNSUBSCRIBE click here: http://topica.com/u/?bUrKDA.bWnbtk.Z2NtYS1h
Or send an email to: [email protected]

TOPICA - Start your own email discussion group. FREE!
http://www.topica.com/partner/tag02/create/index2.html
--^----------------------------------------------------------------