Re: [ITM6] Monitoring Agent for Microsoft SQL Server / Problem Running Multiple Agents

Thanks for the detailed responses.

My observations so far are that there is only a single, common, KOQENV
file.  There are no instance specific environment files.

We have set the KDC_FAMILIES variable in the KOQENV file to add the
ephemeral port.

After restarting a number of instances (remotely via executecommand), only
two have taken the ephemeral port setting, with the rest "ignoring" it.

Agent connected on the ephemeral port
(512D0900.0024-24AC:kbbssge.c,52,"BSS1_GetEnv")
KDE_TRANSPORT=KDC_FAMILIES="IP.SPIPE ephemeral:y PORT:3660 IP use:n SNA
use:n IP.PIPE use:n "
(512D0900.002F-24AC:kdebpop.c,146,"KDEBP_Open") ip.spipe rx/tx/win buffers
set to 4096/4096/8
(512D0900.0031-24AC:kde1otp.c,143,"KDE1I_OpenTransportProvider") Transport
opened: socket/ip.spipe (00C2) 127 1
(512D0900.003C-24AC:kbbssge.c,52,"BSS1_GetEnv")
CT_CMSLIST="IP.SPIPE:#172.17.151.10;IP.SPIPE:#172.25.151.10"
(512D0900.003E-24AC:kraarreg.cpp,2677,"IRA_SetConnectCMSLIST") *INFO: 01
IP.SPIPE:#172.17.151.10
(512D0900.003F-24AC:kraarreg.cpp,2691,"IRA_SetConnectCMSLIST") *INFO:
Primary TEMS set to <IP.SPIPE:#172.17.151.10> host <#172.17.151.10>
(512D0900.00A2-24AC:kdebpap.c,122,"KDEBP_AssignPort") ip.spipe:7756 in
originate-only ephemeral mode
(512D0900.00A3-24AC:kdebpap.c,128,"KDEBP_AssignPort") ip.spipe bound to
port 7756: base=3660, limit=3660
(512D0900.00A4-268C:kdclnew.c,325,"NewSDB") LLB entry 1 is
ip.spipe:#172.16.4.175[3660], local
(512D0900.00A5-268C:kdclnew.c,329,"NewSDB") GLB entry 1 is
ip.spipe:#172.16.4.175[3660], local
(512D0CBD.0000-268C:kraarreg.cpp,346,"ConnectToProxy") Successfully
connected to CMS REMOTE_camkram000mpira using ip.spipe:#172.17.151.10[3660]

This instance did not pick up the ephemeral port setting
(512D3E9D.0024-980:kbbssge.c,52,"BSS1_GetEnv")
KDE_TRANSPORT=KDC_FAMILIES="IP.SPIPE PORT:3660 IP use:n SNA use:n IP.PIPE
use:n"
(512D3E9D.002F-980:kdebpop.c,146,"KDEBP_Open") ip.spipe rx/tx/win buffers
set to 4096/4096/8
(512D3E9D.0030-980:kde1otp.c,143,"KDE1I_OpenTransportProvider") Transport
opened: socket/ip.spipe (00C2) 127 1
(512D3E9D.003B-980:kbbssge.c,52,"BSS1_GetEnv")
CT_CMSLIST="IP.SPIPE:#172.17.151.10;IP.SPIPE:#172.25.151.10"
(512D3E9D.003D-980:kraarreg.cpp,2677,"IRA_SetConnectCMSLIST") *INFO: 01
IP.SPIPE:#172.17.151.10
(512D3E9D.003E-980:kraarreg.cpp,2691,"IRA_SetConnectCMSLIST") *INFO:
Primary TEMS set to <IP.SPIPE:#172.17.151.10> host <#172.17.151.10>
(512D3E9D.00A2-980:kdebpap.c,128,"KDEBP_AssignPort") ip.spipe bound to port
15948: base=3660, limit=3660
(512D3E9D.00A4-1754:kdclnew.c,325,"NewSDB") LLB entry 1 is
ip.spipe:#172.16.4.175[3660], local
(512D3E9D.00A5-1754:kdclnew.c,329,"NewSDB") GLB entry 1 is
ip.spipe:#172.16.4.175[3660], local
(512D3E9D.00A9-1754:kraarreg.cpp,346,"ConnectToProxy") Successfully
connected to CMS REMOTE_camkram000mpira using ip.spipe:#172.17.151.10[3660]


Questions
SHOULD there be instance specific environment files ?
Is the remote agent installation method as reliable as local ?
Is the MTEMS service start up different (less reliable) than services
started from the Services control panel ?




                                                                                      
                                                                                      
                                                                                      
                                                                                      
                                                                                      
   Martin                                         410 Albert Street         (Embedded 
   (M.A.)                                                              image moved to 
   Butler                                                                       file: 
                                                                        pic49569.gif) 
                                                                                      
   Senior IT                                      Waterloo, Ontario                   
   Services                                      N2L 3V3                              
   Professional                                                                       
   - Tivoli on                                                                        
   Demand                                                                             
   Services                                                                           
                                                                                      
   T7Z                                            Canada                              
                                                                                      
   DM                                                                                 
                                                                                      
   Phone:       +1-519-880-2920                                                       
                                                                                      
   e-mail:      [email protected]                                                    
                                                                                      
                                                                                      
                                                                                      
                                                                                      
                                                                                      
                                                                                      
                                                                                      
                                                                                      
                                                                                      
                                                                                      










From:	"[email protected]" <[email protected]>
To:	"Discussion list for Tivoli product and Tivoli Ready products."
            <[email protected]>,
Date:	02/22/2013 10:29 PM
Subject:	Re: [TME10] [ITM6] Monitoring Agent for Microsoft SQL Server /
            Problem Running Multiple Agents
Sent by:	[email protected]



Martin: You have your logical bases covered on this, and I think you're
correct about the root cause being socket contention.  The easy solution
would be to setup "EPHEMERAL:Y" in the KDC_FAMILIES string in each agent
instance's koqenv file--this parameter gets around the max-15-agent
limitation. Thus, if that really is the issue, this would address it. You
could also just specify "EPHEMERAL:inbound" just once in the *ms*config
file of the TEMS these agents connect to.  However, given that there are
some additional configuration items to be wary of using this parameter
(you'll either need to locate the  Warehouse Proxy agent on the TEMS these
agents connect to, or if not, you'll need to use the KDE Gateway feature to
route data exports correctly), I think the following is preferable--plus
it's more interesting from a investigative perspective.

a) Edit each instance's %CANDLE_HOME%\tmaitm6\koqenv file (might be
under ..\tmaitm6_x64) to hardcode the KDC_FAMILIES variable going
sequentially up the SKIP chain and using "COUNT:1" in each as well just to
ensure they only ever try RPC bindings on the one port--we'll place the
COUNT/SKIP parameters to make them apply to IP.SPIPE first since that's
your preferred protocol:

NOTE1: SKIP:1 is the default so we don't need to add it in the first one
NOTE2: In the below lines, both PIPE and SPIPE are enabled, but SPIPE is
always used first to talk to the TEMS

KOQ Agent Instance 1
KDC_FAMILIES=IP.PIPE PORT:1918 IP use:n SNA use:n IP.SPIPE COUNT:1
PORT:3660

KOQ Agent Instance 2
KDC_FAMILIES=IP.PIPE PORT:1918 IP use:n SNA use:n IP.SPIPE COUNT:1 SKIP:2
PORT:3660

KOQ Agent Instance 3
KDC_FAMILIES=IP.PIPE PORT:1918 IP use:n SNA use:n IP.SPIPE COUNT:1 SKIP:3
PORT:3660
....
KOQ Agent Instance 12
KDC_FAMILIES=IP.PIPE PORT:1918 IP use:n SNA use:n IP.SPIPE COUNT:1 SKIP:12
PORT:3660

b) Once complete ensure that not only all KOQ agents, but also any
ITM-related agents/components on the Windows host are stopped.

c) Run the following command from a command prompt and make certain no
lines are returned--we want to be sure nothing is listening on any of the
IP.SPIPE ports that the 12 KOQ agents might occupy.  NOTE: when you pass
'findstr' a series of text in quotation marks, it doesn't treat it as a
literal string but instead treats each space-separate string as a distinct
item to search for, basically OR'ing the whole thing:

netstat -abn | findstr LISTEN | findstr "7756 11852 15948 20044 24140 28236
32332 36428 40524 44620 48716 52812 56908 61004 65100"

d) Beginning with one KOQ agent, start it up from the MTEMS UI.  Wait up to
10 seconds and repeat the command from step c) and ensure you see 7756 as a
LISTENER.  It's okay if you see it twice bound to both "0.0.0.0" and "[::}"

e) Repeat step d) 11 more times for the remaining KOQ agent instances,
ensuring you continue to see the subsequent higher LISTENER port show up in
the netstat output.

f) Login to the TEPS and see whether they have all come online and that you
can view real-time data for each by navigating to a workspace underneath
the particular agent in the navigator tree.

This should work, but if the problem is something other than socket
contention, then we'll just have to goto the koq RAS1 logs for a look.  You
can then start the KNT and custom agents without problem as well, but you
might also want to edit their KDC_FAMILIES lines to add in "COUNT:1
SKIP:13" and "COUNT:1 SKIP:14" to allow their startup to proceed more
quickly, otherwise they'll try each of the first 12 ports before finally
binding to 56908 and 61004, respectively.

Good luck and let us know the result...

Toben

On Fri, Feb 22, 2013 at 5:26 PM, Martin Butler <[email protected]> wrote:

  I have a Windows 2003 (64 bit) server with three varieties of MSSQL
  running, and 17 unique instances, all of which have to be monitored.

  I have a [standard] base OS agent and a custom OS agent installed on this
  server.

  After remotely pushing the SQL agent to 12 instances (SQL 2005 & 2008),
  all
  of which worked perfectly, the thirteenth installation completed but
  never
  showed up on the TEMS/TEPS despite running locally on the server.
  The only way to get this (and any subsequent) agents to connect to the
  hub
  was to change the protocol to ip.pipe  (I want to use ip.spipe for all
  agents if possible)

  I think I am hitting a network port contention (baseport + N*4096)
  although
  I expcected that on agent number 17 (not 15).

  I have since tried stopping/starting agents using the MTEMS and the
  Service
  Control Panel with varying results.  In some cases, changing the protocol
  from ip.spipe to ip.pipe (and back again) resolved my problem. In other
  cases, this had no bearing.

  Of the 12 instance agents that I originally had working, I can now only
  run
  7 fully functional.

  Is it possible that I have locked ports up ?  If so, for how long ?

  I have read about SKIP and COUNT modifiers to control the port search
  algorithm.  Has anyone had success using this on a similar scale ?


  Target server:
  Operating System:  Windows 2003 (64 bit)
  SQL Server 2000 (32 bit)
  SQL Server 2005 (32 & 64 bit)
  SQL Server 2008 (64 bit)


  Agents
  Monitoring Agent for Windows OS  / 06.22.04.00  (64 bit)
  Monitoring Agent for Microsoft SQL Server  / 06.30.00.00


  Appreciate any help.







     Martin                                         410 Albert Street
  (Embedded
     (M.A.)
  image moved to
     Butler
  file:

  pic00325.gif)

     Senior IT                                      Waterloo, Ontario
     Services                                      N2L 3V3
     Professional
     - Tivoli on
     Demand
     Services

     T7Z                                            Canada

     DM

     Phone:       +1-519-880-2920

     e-mail:      [email protected]
















  _______________________________________________
  TME10 mailing list
  [email protected]
  Unsubscribe:[email protected]




--
"Sometimes I think that's the only right thing to do:
To dream. to live in the world of dreams.
But it doesn't last forever--wakefulness always comes to take me back..."
_______________________________________________
TME10 mailing list
[email protected]
Unsubscribe:[email protected]

_______________________________________________
TME10 mailing list
[email protected]
Unsubscribe:[email protected]