Re: Sorting SMF records

Ok, finally I can report to the group thet I've been able to
successfully sort SMF records.

First however, I'd like to thank you all for your input.  Each response
has given me some insight that collectivly have helped me to accomplish
my goal.

Here are my observaions:

    1. I tried to use IFASMFDP to check for invalid records.  That
resulted in complete failure, since the program installed on the TK
systems must be very old and has only one purpose (apparently), and that
is to dump SMF records from a MAN(x) dateset to a sequential dataset
(DUMPOUT) and then clear out the input dataset (DUMPIN).  It does not
accept any SYSIN parameter statements to alter this behavior, and so I
lost some data, because I didn't realize that the input was ALWAYS
cleared out. (No big deal since the whole thing is experimental anyway.)
2. I looked for an alternative on the CBT tape.  I found a newer version
called SMFDUMP, howevr I wasn't able to assemble it, because it is
designed for OS/390 and up, requiring HASM, etc.  I could have tried to
adapt it for my purpose, but I decided not to try.    3. I looked for a
FIXVBS program on the CBT tape, but couldn't find that name.  Instead I
found COPYVBS, which seemed to perform some chchecks on VBS type
recordes, (eg. check if BDW and RDW are acaccuratetly describing the
record/block on the track.)

This was interresting, because at first it seemed to work fine.  (RC =
00, no ABEND).  Then I checked the EXCP stats and found only 1 EXCP had
been used to read a dataset of about 25 tracks.  That seemed suspicious,
and so I printed the output dataset using IDCAMS PRINT.  Only som 28
records had been copied, again without any error message or other
problem indication.  I then used IDCAMS REPRO to copy the input dataset.
That worked and reported some 208 records.

I copied some 40 SMF archive datasets with IDCAMS REPRO.  Some were bad
or empty (because I damaged them by using IFASMFDP on them.) Of the good
ones that were left I created one single archive by concatenating all of
them and feeding them to IDCAMS REPRO.  The result was about 5,108
records.

I still couldn't figure out why COPYVBS woul ony read 1 track (1 EXCP),
so I did some cosmetic changes to the program.  First I added register
equates by using a macro I found in one of Sam Golob's programs
(SAVEALL).  I used this because it also allows for proper save area
chaining, program entry. (and exit using ENDALL).  Then I changed all
the register uses to use the equated register symbolic names. (Now I
could figure out quickly which registers are used in the program.)

Next I added a record counter using R5, and code to format that counter
at the end of the program and print a message.  I decided to count each
record written to the output file.

When running this program on my 5,108  record archive it only copied
about 28 records.  So now  had to figure out why it wouldn't go past
that first track.  It took a while but I found that R10 was used as base
for the input buffer, a well as some other area (which didn't change
after the second EXCP, and therefore the program recognized EOF.  After
making the base for that area R12, the program happily read and wrote
all 5,108 records with no further problems.
    4. Now I tried the SORT again.  I found that the "CORE=" parameter
yields an error message "IER067I - INVALID EXEC OR ATTACH PARAMETER".
The DIAG parameter causes a lot of basically useless messages to be
written to the log, and causes the S0C1 abend if any errors are found. 
If the DIAG parameter is not present the program ends with RC=16.
The last message "IER053A - OUT OF SEQ" is the culprit for the RC=16.
By changing the SORT statement to properly reflect the SMF record fields
of DATE and TIME, I was able to overcome this issue.
  SORT  FIELDS=(19,16,BI,A,11,4,BI,A,7,4,BI,A)
  END
Was my original statement, copied from SYS1.SAMPLIB.  It's wrong!
This is the one I'm using now:
  SORT  FIELDS=(07,04,PD,A,03,4,BI,A)
  END
First field is the record date packed decimal, second is the record time
binary.
This works now, but before the whole sort was successful there were stil
some issues.    5. The SORT program had definite troubles reading
records greater than 7K.  So I reblocked the archive to 7K on a 2314
volume.
The sort program had no trouble writing large records, even larger the
32760. So I moved my sortout to a 3380, because using a 3350 volume
caused some S002-0c, or S002-18 abends.    6. Finally the sort ended
reporting all 5,108 records had been read, and written!

Here is the message log (Ok, by now there are more records i te
archive)!

001  COPYVBS STARTED COPYVBS_09/05/10_17.06
COPYVBS  I/O: WRITTEN:                 6,309 RECORDS
998  COPYVBS ENDED


IER036I - B = 7296
IER037I - G =    52
IER038I - NMAX =     790
IER050I - END MERGE PH
IER054I - RCD IN    6309,OUT    6309
IER052I - EOJ


I verified the sorting sequence using IDCAMS PRINT, and the program
SMFFRMT I found in SYS1.SAMPLIB.

Here is my final SORT JCL:   [:D]

//SMFSORT0     JOB     'ROGER W. SUHR',CLASS=A,MSGCLASS=A,
//             NOTIFY=RWS01
/*JOBPARM LINES=9999
//*-------------------------------------------------------------------
//*
//S005         EXEC    IEFBR14,DS1='SMF.SMFREBLK',
//             DS2='SMF.SMFSORT'
//*
//S010         EXEC    COPYVBS
//SYSUT2       DD      DISP=(,CATLG,DELETE),DSN=SMF.SMFREBLK,
//             DCB=(RECFM=VBS,LRECL=6996,BLKSIZE=7000),
//             UNIT=2314,SPACE=(CYL,(50,50),RLSE),
//             VOL=SER=SORT01
//*
//S020         EXEC    SORT,COND=(0,NE),RGN=6144,
//             PARM='MSG=AP'
//SYSIN        DD      *
  SORT  FIELDS=(07,04,PD,A,03,4,BI,A)
  END
//SORTIN       DD      DISP=SHR,DSN=SMF.SMFREBLK
//SORTOUT      DD      DISP=(,CATLG,DELETE),DSN=SMF.SMFSORT,
//             UNIT=SYSDA,VOL=SER=WORK03,
//             SPACE=(CYL,(50,50),RLSE),
//             DCB=(LRECL=32756,BLKSIZE=32760,RECFM=VB)
//*
//SORDATA      DD      UNIT=2314,SPACE=(CYL,(1,1)),
//             DCB=(RECFM=VBS,LRECL=600,BLKSIZE=200)
//HDRDATA      DD      UNIT=2314,SPACE=(TRK,(5,5)),
//             DCB=(RECFM=VBS,LRECL=600,BLKSIZE=200)
//*
//S990         EXEC    IEFBR14,COND=(0,NE),DS1='SMF.SMFREBLK'

Procs used:

//SORT     PROC  CLS='*',RGN=4096
//*
//SORT     EXEC  PGM=SORT,REGION=&RGN.K,
//         PARM='MSG=AP'
//SYSOUT   DD    SYSOUT=&CLS.
//SYSUDUMP DD    SYSOUT=&CLS.
//SYSPRINT DD    SYSOUT=&CLS.
//SORTLIB  DD    DSNAME=SYS1.SORTLIB,DISP=SHR
//SORTWK01 DD    UNIT=2314,SPACE=(CYL,(5,5))
//SORTWK02 DD    UNIT=2314,SPACE=(CYL,(5,5))
//SORTWK03 DD    UNIT=2314,SPACE=(CYL,(5,5))
//SORTWK04 DD    UNIT=2314,SPACE=(CYL,(5,5))
//SORTWK05 DD    UNIT=2314,SPACE=(CYL,(5,5))
//SORTWK06 DD    UNIT=2314,SPACE=(CYL,(5,5))
//SORTWK07 DD    UNIT=2314,SPACE=(CYL,(5,5))
//SORTWK08 DD    UNIT=2314,SPACE=(CYL,(5,5))

//IEFBR14  PROC DS1=,
//         DS2=,
//         DS3=,
//         DS4=,
//         DS5=
//*
//DEL          EXEC    PGM=IEFBR14
//DD1          DD      DSN=&DS1.,
//             DISP=(MOD,DELETE,DELETE),UNIT=SYSDA,SPACE=(TRK,0)
//DD2          DD      DSN=&DS2.,
//             DISP=(MOD,DELETE,DELETE),UNIT=SYSDA,SPACE=(TRK,0)
//DD3          DD      DSN=&DS3.,
//             DISP=(MOD,DELETE,DELETE),UNIT=SYSDA,SPACE=(TRK,0)
//DD4          DD      DSN=&DS4.,
//             DISP=(MOD,DELETE,DELETE),UNIT=SYSDA,SPACE=(TRK,0)
//DD5          DD      DSN=&DS5.,
//             DISP=(MOD,DELETE,DELETE),UNIT=SYSDA,SPACE=(TRK,0)
//*

//COPYVBS  PROC  G='0'
//*
//COPYVBS  EXEC  PGM=COPYVBS
//*
//STEPLIB  DD    DISP=SHR,DSN=RWS01.LINKLIB
//SYSPRINT DD    SYSOUT=*
//SYSUDUMP DD    SYSOUT=*
//SNAP     DD    SYSOUT=*
//SYSUT1   DD    DISP=SHR,DSN=SMF.DAILY.DATA(&G.)
//SYSUT2   DD    DUMMY,DCB=(RECFM=VBS,LRECL=32756,BLKSIZE=32760)

Thanks again for all your input, it is very much appreciated!



Roger W. Suhr