Re: [ITM6] Monitoring Agent for Microsoft SQL Server / Problem Running Multiple Agents

1) At ITM 623, the best way to proceed is to place any permanent changes 
into an oq.environment file. That should have the same attributes as the 
oq.ini file. Probably the most robust setting would be like this.

KDC_FAMILIES=EPHEMERAL:Y  ${KDC_FAMILIES}

Before I firm up such a change, I always would look at the

logs/oq.env

and see what the exact form is. Particularly - sometimes an "export" is 
required.

After making the change and starting the agent - look at oq.env again... 
make sure it looks right.

2) at ITM 622, you need to create an override file in a same way... and 
then add a source include to the .ini file

.  /opt/IBM/ITM/config/oq.override

In that case the definition needs to be quoted

KDC_FAMILIES="EPHEMERAL:Y  ${KDC_FAMILIES}"

Double quotes allow variable substitution, single quotes not... but quotes 
are always needed.

Other ways are a risky since a maintenance upgrade can change things any 
time. These techniques will survive any maintenance. Here is a technote I 
wrote on the general area. KDC_FAMILIES is always strange because the java 
startup code uses that entry as a place holder and creates the values 
based on defaults. So this sort of override is needed for reliability.

Updating Linux/Unix agent KDC_FAMILIES configuration
http://www.ibm.com/support/docview.wss?uid=swg21390561 

Regards, 

  



John Alvord - Ph: 1-720-396-2788    Cell: none 
Customer Support - Tivoli Software - jalvord-r/[email protected]
Advisory Engineer - Tivoli Monitoring - ITM infrastructure
Follow us on Twitter! @Tivolisupport and Facebook
Personalize your support needs with the IBM Support Portal
Use Service Request to get assistance!
For emails regarding a PMR, copy [email protected]
For secure browser uploads: https://www.ecurep.ibm.com/app/upload
For Customer Support guidance: IBM Software Support Handbook









From:   "[email protected]" <[email protected]>
To:     "Discussion list for Tivoli product and Tivoli Ready products." 
<[email protected]>, 
Date:   02/27/2013 07:58 AM
Subject:        Re: [TME10] [ITM6] Monitoring Agent for Microsoft SQL 
Server /        Problem Running Multiple Agents
Sent by:        [email protected]



Interesting, Martin.  So per what I stated before, another option is using 
"EPHEMERAL:inbound" on the TEMS that these instances connect to.  Try 
adding that to the TEMS' config file (<hostname>_ms_<temsname>.config on 
*NIX and KBBENV on Windows) and recycle the TEMS and see if they show up.

Toben

On Wed, Feb 27, 2013 at 9:30 AM, Martin Butler <[email protected]> wrote:

Thanks for the detailed responses.

My observations so far are that there is only a single, common, KOQENV
file.  There are no instance specific environment files.

We have set the KDC_FAMILIES variable in the KOQENV file to add the
ephemeral port.

After restarting a number of instances (remotely via executecommand), only
two have taken the ephemeral port setting, with the rest "ignoring" it.

Agent connected on the ephemeral port
(512D0900.0024-24AC:kbbssge.c,52,"BSS1_GetEnv")
KDE_TRANSPORT=KDC_FAMILIES="IP.SPIPE ephemeral:y PORT:3660 IP use:n SNA
use:n IP.PIPE use:n "
(512D0900.002F-24AC:kdebpop.c,146,"KDEBP_Open") ip.spipe rx/tx/win buffers
set to 4096/4096/8
(512D0900.0031-24AC:kde1otp.c,143,"KDE1I_OpenTransportProvider") Transport
opened: socket/ip.spipe (00C2) 127 1
(512D0900.003C-24AC:kbbssge.c,52,"BSS1_GetEnv")
CT_CMSLIST="IP.SPIPE:#172.17.151.10;IP.SPIPE:#172.25.151.10"
(512D0900.003E-24AC:kraarreg.cpp,2677,"IRA_SetConnectCMSLIST") *INFO: 01
IP.SPIPE:#172.17.151.10
(512D0900.003F-24AC:kraarreg.cpp,2691,"IRA_SetConnectCMSLIST") *INFO:
Primary TEMS set to <IP.SPIPE:#172.17.151.10> host <#172.17.151.10>
(512D0900.00A2-24AC:kdebpap.c,122,"KDEBP_AssignPort") ip.spipe:7756 in
originate-only ephemeral mode
(512D0900.00A3-24AC:kdebpap.c,128,"KDEBP_AssignPort") ip.spipe bound to
port 7756: base=3660, limit=3660
(512D0900.00A4-268C:kdclnew.c,325,"NewSDB") LLB entry 1 is
ip.spipe:#172.16.4.175[3660], local
(512D0900.00A5-268C:kdclnew.c,329,"NewSDB") GLB entry 1 is
ip.spipe:#172.16.4.175[3660], local
(512D0CBD.0000-268C:kraarreg.cpp,346,"ConnectToProxy") Successfully
connected to CMS REMOTE_camkram000mpira using 
ip.spipe:#172.17.151.10[3660]

This instance did not pick up the ephemeral port setting
(512D3E9D.0024-980:kbbssge.c,52,"BSS1_GetEnv")
KDE_TRANSPORT=KDC_FAMILIES="IP.SPIPE PORT:3660 IP use:n SNA use:n IP.PIPE
use:n"
(512D3E9D.002F-980:kdebpop.c,146,"KDEBP_Open") ip.spipe rx/tx/win buffers
set to 4096/4096/8
(512D3E9D.0030-980:kde1otp.c,143,"KDE1I_OpenTransportProvider") Transport
opened: socket/ip.spipe (00C2) 127 1
(512D3E9D.003B-980:kbbssge.c,52,"BSS1_GetEnv")
CT_CMSLIST="IP.SPIPE:#172.17.151.10;IP.SPIPE:#172.25.151.10"
(512D3E9D.003D-980:kraarreg.cpp,2677,"IRA_SetConnectCMSLIST") *INFO: 01
IP.SPIPE:#172.17.151.10
(512D3E9D.003E-980:kraarreg.cpp,2691,"IRA_SetConnectCMSLIST") *INFO:
Primary TEMS set to <IP.SPIPE:#172.17.151.10> host <#172.17.151.10>
(512D3E9D.00A2-980:kdebpap.c,128,"KDEBP_AssignPort") ip.spipe bound to 
port
15948: base=3660, limit=3660
(512D3E9D.00A4-1754:kdclnew.c,325,"NewSDB") LLB entry 1 is
ip.spipe:#172.16.4.175[3660], local
(512D3E9D.00A5-1754:kdclnew.c,329,"NewSDB") GLB entry 1 is
ip.spipe:#172.16.4.175[3660], local
(512D3E9D.00A9-1754:kraarreg.cpp,346,"ConnectToProxy") Successfully
connected to CMS REMOTE_camkram000mpira using 
ip.spipe:#172.17.151.10[3660]


Questions
SHOULD there be instance specific environment files ?
Is the remote agent installation method as reliable as local ?
Is the MTEMS service start up different (less reliable) than services
started from the Services control panel ?









   Martin                                         410 Albert Street       
  (Embedded
   (M.A.)                                                             
 image moved to
   Butler                                                                 
      file:
                                                                        
pic49569.gif)

   Senior IT                                      Waterloo, Ontario
   Services                                      N2L 3V3
   Professional
   - Tivoli on
   Demand
   Services

   T7Z                                            Canada

   DM

   Phone:       +1-519-880-2920

   e-mail:      [email protected]




















From:   "[email protected]" <[email protected]>
To:     "Discussion list for Tivoli product and Tivoli Ready products."
            <[email protected]>,
Date:   02/22/2013 10:29 PM
Subject:        Re: [TME10] [ITM6] Monitoring Agent for Microsoft SQL 
Server /
            Problem Running Multiple Agents
Sent by:        [email protected]



Martin: You have your logical bases covered on this, and I think you're
correct about the root cause being socket contention.  The easy solution
would be to setup "EPHEMERAL:Y" in the KDC_FAMILIES string in each agent
instance's koqenv file--this parameter gets around the max-15-agent
limitation. Thus, if that really is the issue, this would address it. You
could also just specify "EPHEMERAL:inbound" just once in the *ms*config
file of the TEMS these agents connect to.  However, given that there are
some additional configuration items to be wary of using this parameter
(you'll either need to locate the  Warehouse Proxy agent on the TEMS these
agents connect to, or if not, you'll need to use the KDE Gateway feature 
to
route data exports correctly), I think the following is preferable--plus
it's more interesting from a investigative perspective.

a) Edit each instance's %CANDLE_HOME%\tmaitm6\koqenv file (might be
under ..\tmaitm6_x64) to hardcode the KDC_FAMILIES variable going
sequentially up the SKIP chain and using "COUNT:1" in each as well just to
ensure they only ever try RPC bindings on the one port--we'll place the
COUNT/SKIP parameters to make them apply to IP.SPIPE first since that's
your preferred protocol:

NOTE1: SKIP:1 is the default so we don't need to add it in the first one
NOTE2: In the below lines, both PIPE and SPIPE are enabled, but SPIPE is
always used first to talk to the TEMS

KOQ Agent Instance 1
KDC_FAMILIES=IP.PIPE PORT:1918 IP use:n SNA use:n IP.SPIPE COUNT:1
PORT:3660

KOQ Agent Instance 2
KDC_FAMILIES=IP.PIPE PORT:1918 IP use:n SNA use:n IP.SPIPE COUNT:1 SKIP:2
PORT:3660

KOQ Agent Instance 3
KDC_FAMILIES=IP.PIPE PORT:1918 IP use:n SNA use:n IP.SPIPE COUNT:1 SKIP:3
PORT:3660
....
KOQ Agent Instance 12
KDC_FAMILIES=IP.PIPE PORT:1918 IP use:n SNA use:n IP.SPIPE COUNT:1 SKIP:12
PORT:3660

b) Once complete ensure that not only all KOQ agents, but also any
ITM-related agents/components on the Windows host are stopped.

c) Run the following command from a command prompt and make certain no
lines are returned--we want to be sure nothing is listening on any of the
IP.SPIPE ports that the 12 KOQ agents might occupy.  NOTE: when you pass
'findstr' a series of text in quotation marks, it doesn't treat it as a
literal string but instead treats each space-separate string as a distinct
item to search for, basically OR'ing the whole thing:

netstat -abn | findstr LISTEN | findstr "7756 11852 15948 20044 24140 
28236
32332 36428 40524 44620 48716 52812 56908 61004 65100"

d) Beginning with one KOQ agent, start it up from the MTEMS UI.  Wait up 
to
10 seconds and repeat the command from step c) and ensure you see 7756 as 
a
LISTENER.  It's okay if you see it twice bound to both "0.0.0.0" and 
"[::}"

e) Repeat step d) 11 more times for the remaining KOQ agent instances,
ensuring you continue to see the subsequent higher LISTENER port show up 
in
the netstat output.

f) Login to the TEPS and see whether they have all come online and that 
you
can view real-time data for each by navigating to a workspace underneath
the particular agent in the navigator tree.

This should work, but if the problem is something other than socket
contention, then we'll just have to goto the koq RAS1 logs for a look.  
You
can then start the KNT and custom agents without problem as well, but you
might also want to edit their KDC_FAMILIES lines to add in "COUNT:1
SKIP:13" and "COUNT:1 SKIP:14" to allow their startup to proceed more
quickly, otherwise they'll try each of the first 12 ports before finally
binding to 56908 and 61004, respectively.

Good luck and let us know the result...

Toben

On Fri, Feb 22, 2013 at 5:26 PM, Martin Butler <[email protected]> wrote:

  I have a Windows 2003 (64 bit) server with three varieties of MSSQL
  running, and 17 unique instances, all of which have to be monitored.

  I have a [standard] base OS agent and a custom OS agent installed on 
this
  server.

  After remotely pushing the SQL agent to 12 instances (SQL 2005 & 2008),
  all
  of which worked perfectly, the thirteenth installation completed but
  never
  showed up on the TEMS/TEPS despite running locally on the server.
  The only way to get this (and any subsequent) agents to connect to the
  hub
  was to change the protocol to ip.pipe  (I want to use ip.spipe for all
  agents if possible)

  I think I am hitting a network port contention (baseport + N*4096)
  although
  I expcected that on agent number 17 (not 15).

  I have since tried stopping/starting agents using the MTEMS and the
  Service
  Control Panel with varying results.  In some cases, changing the 
protocol
  from ip.spipe to ip.pipe (and back again) resolved my problem. In other
  cases, this had no bearing.

  Of the 12 instance agents that I originally had working, I can now only
  run
  7 fully functional.

  Is it possible that I have locked ports up ?  If so, for how long ?

  I have read about SKIP and COUNT modifiers to control the port search
  algorithm.  Has anyone had success using this on a similar scale ?


  Target server:
  Operating System:  Windows 2003 (64 bit)
  SQL Server 2000 (32 bit)
  SQL Server 2005 (32 & 64 bit)
  SQL Server 2008 (64 bit)


  Agents
  Monitoring Agent for Windows OS  / 06.22.04.00  (64 bit)
  Monitoring Agent for Microsoft SQL Server  / 06.30.00.00


  Appreciate any help.







     Martin                                         410 Albert Street
  (Embedded
     (M.A.)
  image moved to
     Butler
  file:

  pic00325.gif)

     Senior IT                                      Waterloo, Ontario
     Services                                      N2L 3V3
     Professional
     - Tivoli on
     Demand
     Services

     T7Z                                            Canada

     DM

     Phone:       +1-519-880-2920

     e-mail:      [email protected]
















  _______________________________________________
  TME10 mailing list
  [email protected]
  Unsubscribe:[email protected]




--
"Sometimes I think that's the only right thing to do:
To dream. to live in the world of dreams.
But it doesn't last forever--wakefulness always comes to take me back..."
_______________________________________________
TME10 mailing list
[email protected]
Unsubscribe:[email protected]

_______________________________________________
TME10 mailing list
[email protected]
Unsubscribe:[email protected]




-- 
"Sometimes I think that's the only right thing to do:
To dream. to live in the world of dreams.
But it doesn't last forever--wakefulness always comes to take me back..." 
_______________________________________________
TME10 mailing list
[email protected]
Unsubscribe:[email protected]

_______________________________________________
TME10 mailing list
[email protected]
Unsubscribe:[email protected]