RE: [AM] [dm-discuss] ZZZ Farewell Fellow Agilists

My comments in line, prefixed with DG. I also put a few headings in (this is
getting to be more like a paper than an email.) If you are going to read the
whole thing, this may be a good time to get a cup of coffee. It also took
while due to distractions (work) having to be done...

> -----Original Message-----
> From: Paul Oldfield [mailto:[email protected]
<mailto:[email protected]> ]
> Sent: 02 February 2004 16:38
> To: INTERNET:[email protected]
> Subject: RE: [AM] [dm-discuss] ZZZ Farewell Fellow Agilists
> 
> 
> (responding to Dagna)

<snip>

WHY WAS Y2K SO HARD?

> > Remember Y2K?
> 
> Indeed.  I didn't hear of any really bad horror stories,
> but loads of potential for things to go wrong.  What I'd
> like to do is take that as a driver for making that sort of change
> easier.  Why was it so hard?  Can we do things now to make
> any future change of that sort easier?

DG: It was hard because there is a huge amount of very old code still in
use, which was originally written when storage was expensive. Someone
(Martin Butler?) calculated that the cost of the Y2K changes was less than
the extra storage would have cost over the years if the dates had been
written with four-digit years in the 60s and 70s. (I think the calculation
also involved Present Value, which I was taught in accounting classes way
back when PASCAL was new.)

LEGACY SYSTEMS ARE THE ONES YOU WISH YOU COULD LEAVE BEHIND YOU

> 
> > It is a fairly extreme example - we had to find and change
> a load of
> > columns, and then test everything. And have people standing
> by in case
> > anything got missed.
> 
> I'm not sure I understand the full ramifications of the
> problem, but it seems to me that a large part of the problem 
> in this case was the failure to uncouple the meaning of the 
> data from the implementation.  Applications are going 
> straight into the database rather than going through an 
> interface. Say, using embedded SQL rather than stored 
> procedures. (No doubt there are other ways to provide such 
> an interface).

DG: All the things that seem so obvious and easy now were once pretty
revolutionary. Consider that most companies do the important and/or time
consuming processes first, so the stuff the business really depends on
(order processing, producing bills, recording customers) is often the
oldest. Replacing these systems is a scary prospect - they hold business
rules and processes as well as data - so new developments have to fit into
the existing landscape. Working out what there is, how it fits together,
what it all means, and how to insert new stuff in so that the ugly joins
(transformations) are minimised, is what keeps me in gainful employment. I
have the impression that few (if any) new companies bother with a formal
data or application architecture - they are small enough that they know how
it all fits together, and they only have one billing system, order
processing system, HR system... Ten years and/or a few takeovers and mergers
on, the picture is very different, and the Architecture Team appears in a
corner somewhere. (A nice, quiet team, no trouble to anyone. Trust me, I'm
an Architect!)

Two examples. I changed employers in 1999; my old employers had a major
system (one of two that the company totally depended on) which celebrated
its 21st birthday in about 1995. It was written in Assembler. My new
employers had a major system that stored its data on magnetic tape. (That
isn't this employer, BTW). The data from both of these systems got sent all
over the place, and any change to either could have an impact on the rest of
their systems. Even a small impact per system adds up when you hit a high
enough percentage of the 120-odd systems you have. (I have a theory that 120
is a magic number - having worked at three large companies all with about
that many major applications.)

A lot of these old systems don't have databases - they have files. They
don't always have indexes. Updates are FUN. (Especially when the storage
medium is absolutely serial - mag tape.)

I think that Y2K inspired many companies to bite the bullet and replace
their old systems with the modern, relational or OO version. But my
impression is that most companies didn't - when a system has been in place
for ten years or more, maintained in house, then the likelihood that anyone
knows everything it does and what, exactly, each little wrinkle in the code
is for, is slim. Even if the documentation exists, it can be tough to put
all the pieces together and work out what the new one needs to do and how
the data is migrated. My first warehouse project, one of the two source
systems was a fifteen year old, in-house, major system. To be sure that we
knew how it worked, we had an experienced analyst have her own copy and put
individual transactions through and check all the data files after every
program in every batch to see what it did. As she found things, I modelled
them and then we talked to the developers and said 'is that right' - a lot
of the stuff was things that had been forgotten over the years. We had
documentation going back to the original file layout sheets, and the
original developers were still with the company (one was head of IT), and we
had very experienced developers working on the system and doing the
analysis. It still took us a year - two people, one full time, one most of
the time. (Experiences like this are, I think, what make some of us old
hacks nervous about Agile methods, or anything else that seems to promise
speed - we are afraid for the documentation and have too many grey hairs
gained in maintenance over the years.)

Encapsulation can fix all this! The database gets hidden, and you just see
the API... So you go and see the users, and tell them that you want to
encapsulate your major databases, and that it will cost x thousand/million
and they won't notice the difference. You have to have pretty clued up users
if they are going to see that this should be done before their new toy
(sorry - tool) gets bought, and that it is worth doing something that they
won't notice. Or you can try to steer the conversation so they first see the
problem, then they ask 'what can be done'. This will take time (days or
weeks, often years). If you can get them sold on the scale of the problem,
then getting buy-in for encapsulation is much easier - it is way cheaper
than redeveloping the original app. (Based on gut feeling rather than
metrics).

EVERYONE NEEDS TO KNOW EVERYTHING!

> Anyway, the basic problem from my point of view is that
> I'm not sure I understand.  I may be good at 'agility' but I
> am not a DBA.

DG: Unless you need to know how the DBMS works, you don't need to spend too
much time discovering the fine details (although a general understanding is
essential, IMHO). That's what the DBAs are paid for - and part of their job
should be helping people who don't need to know the details, work stuff out
that they need to do. I firmly believe that what we all need is the ability
to understand the other person's concerns and viewpoint. Even if we don't
agree with it (it makes arguing and persuading much easier). Next time a DBA
has a hissy fit and refuses point blank to even consider doing x - ask them
why, and explain that if you understand the reasons it will make both your
lives much easier in future. And when they have explained, you can maybe
rephrase your request or redo it in a way that won't bring western
civilisation down by the end of the day. (As the aforementioned DBA
explained would be the inevitable consequence of your new column.)

And, sometimes, someone is horribly overworked, incompetent, in a bad mood,
lazy, or just hates you. 

> 
> > But any change to a database has to be checked to see
> > what else uses it - other apps, interfaces, user reporting tools
> > (Business Objects, Excel, Access...) And if the DBA misses one, the 
> > project that asked for the change probably won't get users phoning 
> > them up and screaming down the phone... Which is one reason 
> why they
> > don't like changing columns 'on the fly'.
> 
> My thoughts are that if everyone went through stored
> procedures, then you'd only have them as clients of the 
> database.  I guess the picture isn't so simple for some 
> reason, but why not?

DG: If every new development uses stored procedures, then life will get
easier. And, eventually, all the old apps will get replaced (the tape drives
will get too expensive to maintain, the people who write assembler will come
up to retirement and no-one will want to take a job using it, the companies
will get taken over and the either the take overer or the take overee will
have a spiffy new system and there will be a corporate will to move from a
dozen billing systems to only two or three, and then the old data will get
migrated or dumped. And there will be world peace. <g>)

And one day, an old hack will be someone who remembers Java. (Currently, an
'old hack' is likely to be someone who remembers paper tape, punch cards,
and that you could play Smoke on the Water on mainframe core memory.)

DATA WAREHOUSING

<snip>

> > I have been involved in four data
> > warehouse projects - two built, one abandoned when the company was 
> > sold, one may still be going on as it was for the company 
> that sold my
> > then employers.
> 
> Does Data Warehousing have special problems?  Again, I
> don't know enough to comment on what approaches would be good.

DG: Yes. The big special problem of data warehousing is data quality. Things
like: 

1. Name has different lengths, structures and contents on every system -
some have surname/initials, some have free-format text, lengths vary from 20
to 1024 characters (hey, it's a Varchar, it won't use what it doesn't need,
so let's be safe!), so you need to look at each one and work out how to
transform it and what to do when it fails (name is blank, for example). Then
there are addresses. Always a source of fun for the ETL programmers.

2. Identifiers aren't unique across systems, so they all have to be
converted to a unique Warehouse id. And some of the source systems have
different id formats (number, alphanumeric, formatted
alpha/number/alpha...). And the same customer can have different ids on
different systems and there is no cross-reference to identify the matches.

3. What do you do with the rubbish? Stick it in anyway, or reject it? And
how does it get fixed, in either case? These questions have to be answered
for every table. Different tables may have different answers. (This is
emphatically not a development team decision. It has to be made by the
users, who have to realise the consequences of their decision. Resist all
attempts by the users to push it back to you.)

The other warehouse problem is the size of the thing. The first iteration
will take a long time (whatever value of 'long' you use), because that's
where you set things up so the next bits just add in and use the
infrastructure (way of generating ids, links back to source systems, etl
tool, database instance, etc) that you decided on and set up in phase 1.

If you are developing a warehouse, you need a full time data
analyst/modeller/architect on the project who knows the data, knows why it
was done like that, knows what can be changed (so when you want to change a
format, they either say 'OK', 'OK but...', or threaten you with extreme
violence), and knows what transformations are needed, have been done, or are
near enough like the one you are starting that you can re-use the old one,
and they know who to go to when the feed you have finished, tested,
installed and are running live is suddenly changed. (Because you are
re-using existing interface files and someone, somewhere, needed a change
made. You can probably sell tickets to this meeting, to anyone who isn't too
squeamish.)

> 
> > I joined this list to find out more about Agile methods,
> and see where
> > Enterprise Architecture fits in (I think I know, and when I
> am sure, I
> > will post my ideas).
> 
> Always good to have another point of view.  If you don't think
> we're talking sense, yell.  Somebody will learn something,
> with any luck.  Either we'll learn to express our ideas better,
> or we'll learn to express better ideas   ;-)

DG: Thanks. I will try to keep future replies  a little shorter.


Regards,

Dagna
Dagna Gaythorpe
Data Architect
International IT
> tttt COLT TELECOM GROUP PLC
> Beaufort House, 15 St Botolph Street,
> London EC3A 7QN
t   (+ 44) 020 7 390 7896
f   (+ 44) 020 7 947 1176
e [email protected]




*************************************************************************************
COLT Telecommunications
Registered in England No. 2452736
Registered Office: Beaufort House, 15 St. Botolph Street, London, EC3A 7QN
Tel. +44 20 7390 3900


This message is subject to and does not create or vary any contractual
relationship between COLT Telecommunications, its subsidiaries or 
affiliates ("COLT") and you. Internet communications are not secure
and therefore COLT does not accept legal responsibility for the
contents of this message.  Any view or opinions expressed are those of
the author. The message is intended for the addressee only and its
contents and any attached files are strictly confidential. If you have
received it in error, please telephone the number above. Thank you.
*************************************************************************************

For more information about AM, visit the Agile Modeling Home Page at www.agilemodeling.com
--^----------------------------------------------------------------
This email was sent to: [email protected]

EASY UNSUBSCRIBE click here: http://topica.com/u/?bUrKDA.bWnbtk.Z2NtYS1h
Or send an email to: [email protected]

TOPICA - Start your own email discussion group. FREE!
http://www.topica.com/partner/tag02/create/index2.html
--^----------------------------------------------------------------