Re: Splunk Integration Question...
Sean Alderman <[email protected]> Fri, 13 Sep 2013 09:55:34 -0400
| Newsgroups | gmane.network.nagios.user |
|---|---|
| Message-ID | <CAN39mj2vjY12apMXiYXUKNTy6awndWPr1dKEUw9bUuHZfF644w@mail.gmail.com> |
--===============6928900659202648495==
Content-Type: multipart/alternative; boundary=089e013d1eaa5e3fc804e64435ec
--089e013d1eaa5e3fc804e64435ec
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
>From what I can tell, after trying it, the query string appended to the
splunk_url parameter referneces Nagios specific things...
e.g. https://*splunk_url*/?q=3Dsearch?%20*hostname*%20*
Nagios_command_description
*
So, the implication is that somehow splunk has data about nagios checks, by
name.
For my environment, splunk uses short names. We set this up to avoid
having a mix of both short names and fqdns in the host field since most of
our splunk data is sourced from syslog which doesn't provide fqdn. Because
of this clicking the link for this in Nagios would result in splunk on the
FQDN and producing no results.
The other issue is that at this point, splunk has no data about nagios
checks, so searching the check name, also does nothing for us. So I hope
you can see why I'm confused about the purpose of this integration.
On Tue, Sep 10, 2013 at 3:12 PM, Frost, Mark {BIS}
<[email protected]>wrote:
> Huh. Where did those new options come from? They weren=92t in the
> cgi.cfg docs the last time I looked J.****
>
> ** **
>
> I agree, it=92s not terribly clear to me what that option does, but it do=
es
> reference =93Splunk IT=94 which is a special Splunk package that you can =
use
> for Splunk benchmarking. That still doesn=92t make it clear what it=92s=
used
> for.****
>
> ** **
>
> I see a second parameter, =93splunk_url=94 that lets you specify the URL =
for
> your Splunk server.****
>
> ** **
>
> Maybe it just somehow says to pepper the logs with your Splunk URL in
> appropriate places.****
>
> ** **
>
> Mark****
>
> ** **
>
> *From:* Sean Alderman [mailto:[email protected]]
> *Sent:* Tuesday, September 10, 2013 1:34 PM
> *To:* Nagios Users List
>
> *Subject:* Re: [Nagios-users] Splunk Integration Question...****
>
> ** **
>
> Just what's in the nagios doc on CGI.cfg. The doc is lacking about what i=
t
> does, so I guess I'm a little curious what that config is about.****
>
> - Sean Alderman
> Senior Engineer, UDit Systems Integration****
>
> This message had been brought to you by Android Bionic.****
>
> On Sep 10, 2013 1:10 PM, "Frost, Mark {BIS}" <[email protected]>
> wrote:****
>
> Sean,****
>
> ****
>
> Can you describe what you=92re doing for Splunk integration with Nagios?
> I=92ve used Splunk with Nagios in a couple different ways, but I=92m not =
aware
> of any single standard for doing so.****
>
> ****
>
> Originally, I just had Splunk run a scheduled search, which would trigger
> a script which sent a passive check result back to a Nagios service via
> NSCA. That way =96 having Nagios process passive check results from Spl=
unk
> =96 was the only way I could see to do that.****
>
> ****
>
> Recently, I played around a bit with writing scripts that made use of
> Splunk=92s REST API so the checks could be run as active checks from Nagi=
os.
> (I always prefer active checks). I set this up for only one check, but
> once I got it working it worked pretty well.****
>
> ****
>
> As a side note, I=92m still a little on the fence about whether or not I
> really want to have Nagios find problems through Splunk and then alert on
> them or have Splunk find an alert on them directly without using Nagios a=
t
> all=85****
>
> ****
>
> Are you referring to another way of making Splunk and Nagios talk togethe=
r?
> ****
>
> ****
>
> Mark****
>
> ****
>
> *From:* Sean Alderman [mailto:[email protected]]
> *Sent:* Monday, September 09, 2013 1:12 PM
> *To:* [email protected]
> *Subject:* [Nagios-users] Splunk Integration Question...****
>
> ****
>
> Greetings,****
>
> I was hoping I might find someone who's got the splunk integration
> actively working. I'm running Nagios Core (via EPEL) and Splunk 5.0.3 on
> OracleLinux 6.4.****
>
> When I edit cgi.cfg and enable splunk integration, then set the splunk
> URL to https://<mysplunkserver>:8000/en-US/app/search/flastimeline, I
> notice the nagios URLs look like: https://
> <mysplunkserver>:8000/en-US/app/flashtimeline?q=3Dsearch%20test1.udayton.=
edu%20<nagios
> plugin check>. I have two questions...****
>
> =B7 Is there a way I can make nagios use the hostname only, not t=
he
> FQDN? We use short names in splunk so we don't a mix of fqdn and short
> names since we use both forwarders and syslog as input.****
>
> =B7 What data is this query looking for, is it expected that I
> should have my nagios log in splunk? The <nagios plugin check> in the
> query doesn't seem useful to me, unless there's splunk data specifically
> tied to that check, and I'm hoping someone could provide an example.
> ****
>
> Kind regards,****
>
> -- ****
>
> Sean M. Alderman
> Senior Engineer, UDit Systems Integration and Engineering
> University of Dayton****
>
>
>
> -------------------------------------------------------------------------=
-----
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=3D51271111&iu=3D/4140/ostg.=
clktrk
> _______________________________________________
> Nagios-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nagios-users
> ::: Please include Nagios version, plugin version (-v) and OS when
> reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null***=
*
>
>
> -------------------------------------------------------------------------=
-----
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=3D51271111&iu=3D/4140/ostg.=
clktrk
> _______________________________________________
> Nagios-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nagios-users
> ::: Please include Nagios version, plugin version (-v) and OS when
> reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null
>
--=20
Sean M. Alderman
Senior Engineer, UDit Systems Integration and Engineering
University of Dayton
300 College Park
Dayton, Ohio 45469-1530
(937) 229-5088
[email protected]
*"We are not some casual and meaningless product of evolution. Each of us
is the result of a thought of God. Each of us is willed. Each of us is
loved. Each of us is necessary."* - BXVI
--089e013d1eaa5e3fc804e64435ec
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div><div>From what I can tell, after trying it, the =
query string appended to the splunk_url parameter referneces Nagios specifi=
c things...<br><br></div>e.g.=A0 https://<b>splunk_url</b>/?q=3Dsearch?%20<=
b>hostname</b>%20<b>Nagios_command_description<br>
<br></b></div>So, the implication is that somehow splunk has data about nag=
ios checks, by name.<br><br>For my environment, splunk uses short names.=A0=
We set this up to avoid having a mix of both short names and fqdns in the =
host field since most of our splunk data is sourced from syslog which doesn=
't provide fqdn.=A0 Because of this clicking the link for this in Nagio=
s would result in splunk on the FQDN and producing no results.<br>
<br></div>The other issue is that at this point, splunk has no data about n=
agios checks, so searching the check name, also does nothing for us.=A0 So =
I hope you can see why I'm confused about the purpose of this integrati=
on.<br>
</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Tue,=
Sep 10, 2013 at 3:12 PM, Frost, Mark {BIS} <span dir=3D"ltr"><<a href=
=3D"mailto:[email protected]" target=3D"_blank">[email protected]=
om</a>></span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<div link=3D"blue" vlink=3D"purple" lang=3D"EN-US">
<div>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d">Huh.=A0=A0 Where did thos=
e new options come from?=A0 They weren=92t in the cgi.cfg docs the last tim=
e I looked
</span><span style=3D"font-size:11.0pt;font-family:Wingdings;color:#1f497d"=
>J</span><span style=3D"font-size:11.0pt;font-family:"Calibri",&q=
uot;sans-serif";color:#1f497d">.<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d"><u></u>=A0<u></u></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d">I agree, it=92s not terri=
bly clear to me what that option does, but it does reference =93Splunk IT=
=94 which is a special Splunk package that you can use for Splunk
benchmarking. =A0=A0That still doesn=92t make it clear what it=92s used fo=
r.<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d"><u></u>=A0<u></u></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d">I see a second parameter,=
=93splunk_url=94 that lets you specify the URL for your Splunk server.<u><=
/u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d"><u></u>=A0<u></u></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d">Maybe it just somehow say=
s to pepper the logs with your Splunk URL in appropriate places.<u></u><u><=
/u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d"><u></u>=A0<u></u></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d">Mark<u></u><u></u></span>=
</p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca=
libri","sans-serif";color:#1f497d"><u></u>=A0<u></u></span><=
/p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><b><span style=3D"font-si=
ze:10.0pt;font-family:"Tahoma","sans-serif"">From:</spa=
n></b><span style=3D"font-size:10.0pt;font-family:"Tahoma","=
sans-serif""> Sean Alderman [mailto:<a href=3D"mailto:salderman1@udayt=
on.edu" target=3D"_blank">[email protected]</a>]
<br>
<b>Sent:</b> Tuesday, September 10, 2013 1:34 PM<br>
<b>To:</b> Nagios Users List</span></p><div class=3D"im"><br>
<b>Subject:</b> Re: [Nagios-users] Splunk Integration Question...<u></u><u>=
</u></div><p></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><u></u>=A0<u></u></p>
<p style=3D"margin-left:.5in">Just what's in the nagios doc on CGI.cfg.=
The doc is lacking about what it does, so I guess I'm a little curious=
what that config is about.<u></u><u></u></p><div><div class=3D"h5">
<p style=3D"margin-left:.5in">- Sean Alderman <br>
Senior Engineer, UDit Systems Integration<u></u><u></u></p>
<p style=3D"margin-left:.5in">This message had been brought to you by Andro=
id Bionic.<u></u><u></u></p>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">On Sep 10, 2013 1:10 PM, =
"Frost, Mark {BIS}" <<a href=3D"mailto:[email protected]=
" target=3D"_blank">[email protected]</a>> wrote:<u></u><u></u></p=
>
<div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">Sean,</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">Can you describe what you=92re doing for Splunk =
integration with Nagios?=A0=A0 I=92ve used Splunk with Nagios in a couple d=
ifferent ways, but I=92m not aware of any single standard for doing
so.</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">Originally, I just had Splunk run a scheduled se=
arch, which would trigger a script which sent a passive check result back t=
o a Nagios service via NSCA.=A0=A0 That way =96 having Nagios
process passive check results from Splunk =96 was the only way I could see=
to do that.</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">Recently, I played around a bit with writing scr=
ipts that made use of Splunk=92s REST API so the checks could be run as act=
ive checks from Nagios.=A0 (I always prefer active checks).=A0=A0
I set this up for only one check, but once I got it working it worked pret=
ty well.</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">As a side note, I=92m still a little on the fenc=
e about whether or not I really want to have Nagios find problems through S=
plunk and then alert on them or have Splunk find an alert
on them directly without using Nagios at all=85</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">Are you referring to another way of making Splun=
k and Nagios talk together?</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">Mark</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:.5in">
<span style=3D"font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1f497d">=A0</span><u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in">
<b><span style=3D"font-size:10.0pt;font-family:"Tahoma","san=
s-serif"">From:</span></b><span style=3D"font-size:10.0pt;font-family:=
"Tahoma","sans-serif""> Sean Alderman [mailto:<a href=
=3D"mailto:[email protected]" target=3D"_blank">[email protected]=
</a>]
<br>
<b>Sent:</b> Monday, September 09, 2013 1:12 PM<br>
<b>To:</b> <a href=3D"mailto:[email protected]" target=3D"=
_blank">[email protected]</a><br>
<b>Subject:</b> [Nagios-users] Splunk Integration Question...</span><u></u>=
<u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in">
=A0<u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in">
Greetings,<u></u><u></u></p>
</div>
<p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt;margin-left:1.0in">
=A0 I was hoping I might find someone who's got the splunk integration =
actively working.=A0 I'm running Nagios Core (via EPEL) and Splunk 5.0.=
3 on OracleLinux 6.4.<u></u><u></u></p>
</div>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in">
=A0=A0 When I edit cgi.cfg and enable splunk integration, then set the splu=
nk URL to <a href=3D"https://%3cmysplunkserver%3e:8000/en-US/app/search/fla=
stimeline" target=3D"_blank">
https://<mysplunkserver>:8000/en-US/app/search/flastimeline</a>, I no=
tice the nagios URLs look like: https://<mysplunkserver>:8000/en-US/a=
pp/flashtimeline?q=3Dsearch%<a href=3D"http://20test1.udayton.edu" target=
=3D"_blank">20test1.udayton.edu</a>%20<nagios plugin
check>.=A0 I have two questions...<u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:1.5in">
<span style=3D"font-size:10.0pt;font-family:Symbol">=B7</span><span style=
=3D"font-size:7.0pt">=A0=A0=A0=A0=A0=A0=A0=A0
</span>Is there a way I can make nagios use the hostname only, not the FQDN=
?=A0 We use short names in splunk so we don't a mix of fqdn and short n=
ames since we use both forwarders and syslog as input.<u></u><u></u></p>
<p class=3D"MsoNormal" style=3D"margin-left:1.5in">
<span style=3D"font-size:10.0pt;font-family:Symbol">=B7</span><span style=
=3D"font-size:7.0pt">=A0=A0=A0=A0=A0=A0=A0=A0
</span>What data is this query looking for, is it expected that I should ha=
ve my nagios log in splunk?=A0 The <nagios plugin check> in the query=
doesn't seem useful to me, unless there's splunk data specifically=
tied to that check, and I'm hoping someone could
provide an example.<br clear=3D"all">
<u></u><u></u></p>
</div>
<div>
<div>
<div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in">
Kind regards,<u></u><u></u></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in">
-- <u></u><u></u></p>
<div>
<p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt;margin-left:1.0in">
Sean M. Alderman<br>
Senior Engineer, UDit Systems Integration and Engineering<br>
University of Dayton<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><br>
---------------------------------------------------------------------------=
---<br>
How ServiceNow helps IT people transform IT departments:<br>
1. Consolidate legacy IT systems to a single system of record for IT<br>
2. Standardize and globalize service processes across IT<br>
3. Implement zero-touch automation to replace manual, redundant tasks<br>
<a href=3D"http://pubads.g.doubleclick.net/gampad/clk?id=3D51271111&iu=
=3D/4140/ostg.clktrk" target=3D"_blank">http://pubads.g.doubleclick.net/gam=
pad/clk?id=3D51271111&iu=3D/4140/ostg.clktrk</a><br>
_______________________________________________<br>
Nagios-users mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Nag=
[email protected]</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/nagios-users" targe=
t=3D"_blank">https://lists.sourceforge.net/lists/listinfo/nagios-users</a><=
br>
::: Please include Nagios version, plugin version (-v) and OS when reportin=
g any issue.<br>
::: Messages without supporting info will risk being sent to /dev/null<u></=
u><u></u></p>
</div>
</div></div></div>
</div>
<br>-----------------------------------------------------------------------=
-------<br>
How ServiceNow helps IT people transform IT departments:<br>
1. Consolidate legacy IT systems to a single system of record for IT<br>
2. Standardize and globalize service processes across IT<br>
3. Implement zero-touch automation to replace manual, redundant tasks<br>
<a href=3D"http://pubads.g.doubleclick.net/gampad/clk?id=3D51271111&iu=
=3D/4140/ostg.clktrk" target=3D"_blank">http://pubads.g.doubleclick.net/gam=
pad/clk?id=3D51271111&iu=3D/4140/ostg.clktrk</a><br>___________________=
____________________________<br>
Nagios-users mailing list<br>
<a href=3D"mailto:[email protected]">[email protected]=
urceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/nagios-users" targe=
t=3D"_blank">https://lists.sourceforge.net/lists/listinfo/nagios-users</a><=
br>
::: Please include Nagios version, plugin version (-v) and OS when reportin=
g any issue.<br>
::: Messages without supporting info will risk being sent to /dev/null<br><=
/blockquote></div><br><br clear=3D"all"><br>-- <br><div dir=3D"ltr">Sean M.=
Alderman<br>Senior Engineer, UDit Systems Integration and Engineering<br>
University of Dayton<br>300 College Park<br>Dayton, Ohio 45469-1530<br><a>(=
937) 229-5088</a><br><a href=3D"mailto:[email protected]" target=3D"_b=
lank">[email protected]</a><br><br><i>"We are not some casual and=
meaningless product of evolution. Each=20
of us is the result of a thought of God. Each of us is willed. Each of=20
us is loved. Each of us is necessary."</i>=A0 - BXVI<br><br></div>
</div>
--089e013d1eaa5e3fc804e64435ec--
--===============6928900659202648495==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. Consolidate legacy IT systems to a single system of record for IT
2. Standardize and globalize service processes across IT
3. Implement zero-touch automation to replace manual, redundant tasks
http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
--===============6928900659202648495==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Nagios-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
::: Messages without supporting info will risk being sent to /dev/null
--===============6928900659202648495==--