RE: Open Source centralized log management/SIEM solutions

"Starks, Michael" <[email protected]> Tue, 27 Apr 2010 11:56:00 -0500
Newsgroups gmane.comp.sysutils.loganalysis
Message-ID <3DEF3AA3117A5348BE4D0B292CDA6AFF01EE5FAB@AOEXCH01.namerica.NA.INT.ATOSORIGIN.COM> 
This is a multi-part message in MIME format.

--===============1802345070==
Content-class: urn:content-classes:message
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CAE62A.847244B0"

This is a multi-part message in MIME format.

------_=_NextPart_001_01CAE62A.847244B0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I believe OSSEC, plus Splunk would be a great low/no cost solution which
would meet all of your requirements. Syslog-ng can also be bolted on if
you want something like file system-level log storage with directories
based on hostnames.
=20
I somewhat agree with the quote below if by "home grown" solutions,
Gartner means some internally developed scripts doing some string/regex
matching. If, on the other hand, they also include a collection of
mature open source tools to put the pieces together, I do not
necessarily think it's on the mark. Many of today's open-source log
management/analysis solutions rival the expensive commercial offerings
and provide a lot of value for the investment.

________________________________

From: [email protected]
[mailto:[email protected]] On Behalf Of Youngquist,
Jason R.
Sent: Monday, April 26, 2010 10:04 AM
To: '[email protected]'
Subject: [logs] Open Source centralized log management/SIEM solutions



Is anyone using any Open Source or low cost centralized log
management/SIEM solution in a production environment which you would
recommend?

=20

Specifically, I'm looking for:

--scalability - must be able to handle hundreds of log sources -
majority being servers and network devices

--good searching capability

--ability to generate alerts

--good reporting capability - pre-built reports would be nice

--a solution auditors would approve

--able to meet regulatory requirements such as PCI

--fast implementation time - how long would it take to get the solution
up and running?

=20

=20

There are more things I'd like, but these are the big requirements.

=20

=20

If an Open Source solution, are there any companies that offer
professional services (ie. consulting/configuration assistance) so we
could hit the ground running and not have to spend weeks/months
configuring/creating rules/reports, etc.  Ideally, the solution should
have some commercial support behind it so if we run into any issues we
can speak to a knowledgeable person.

=20

=20

For those QSAs out there, are there any Open Source solutions/low-cost
solutions that you have seen implemented well and meet the PCI
regulatory guidelines?  If so, what were they?  If not, what were they
lacking that commercial products provide?

=20

=20

For those of you with a home-grown/Open Source log management solution,
do you agree with the Gartner quote below?  Why/why not? =20

According to Gartner researchers, "Although [home-grown log management]
may prove effective for a limited set of data sources with clearly
defined "strings" that the organization is searching for, most
organizations quickly run into scalability issues, as well as issues
using the data for situational awareness in support of incident
response. In most cases, internally developed centralized application
log solutions will fall short of meeting organizational requirements."

=20

If you had to do it again would you "roll your own solution" or purchase
a commercial log management product?=20

=20

=20

Appreciate any information you can provide.

=20

=20

Thanks.

Jason Youngquist

Information Technology Security Engineer, Security+

Technology Services

Columbia College

1001 Rogers Street, Columbia, MO  65216

(573) 875-7334

[email protected]

http://www.ccis.edu

=20

=20


------_=_NextPart_001_01CAE62A.847244B0
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns=3D"http://www.w3.org/TR/REC-html40" xmlns:v =3D=20
"urn:schemas-microsoft-com:vml" xmlns:o =3D=20
"urn:schemas-microsoft-com:office:office" xmlns:w =3D=20
"urn:schemas-microsoft-com:office:word" xmlns:x =3D=20
"urn:schemas-microsoft-com:office:excel" xmlns:p =3D=20
"urn:schemas-microsoft-com:office:powerpoint" xmlns:a =3D=20
"urn:schemas-microsoft-com:office:access" xmlns:dt =3D=20
"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s =3D=20
"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs =3D=20
"urn:schemas-microsoft-com:rowset" xmlns:z =3D "#RowsetSchema" xmlns:b =
=3D=20
"urn:schemas-microsoft-com:office:publisher" xmlns:ss =3D=20
"urn:schemas-microsoft-com:office:spreadsheet" xmlns:c =3D=20
"urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc =3D=20
"urn:schemas-microsoft-com:office:odc" xmlns:oa =3D=20
"urn:schemas-microsoft-com:office:activation" xmlns:html =3D=20
"http://www.w3.org/TR/REC-html40" xmlns:q =3D=20
"http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc =3D=20
"http://microsoft.com/officenet/conferencing" XMLNS:D =3D "DAV:" =
XMLNS:Repl =3D=20
"http://schemas.microsoft.com/repl/" xmlns:mt =3D=20
"http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2 =3D=20
"http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda =3D=20
"http://www.passport.com/NameSpace.xsd" xmlns:ois =3D=20
"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir =3D=20
"http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds =3D=20
"http://www.w3.org/2000/09/xmldsig#" xmlns:dsp =3D=20
"http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc =3D=20
"http://schemas.microsoft.com/data/udc" xmlns:xsd =3D=20
"http://www.w3.org/2001/XMLSchema" xmlns:sub =3D=20
"http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec =
=3D=20
"http://www.w3.org/2001/04/xmlenc#" xmlns:sp =3D=20
"http://schemas.microsoft.com/sharepoint/" xmlns:sps =3D=20
"http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi =3D=20
"http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs =3D=20
"http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf =3D=20
"http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p =3D=20
"http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf =3D=20
"http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss =3D=20
"http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi =3D=20
"http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi =3D=20
"http://schemas.openxmlformats.org/package/2006/digital-signature" =
xmlns:mver =3D=20
"http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m =
=3D=20
"http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels =3D=20
"http://schemas.openxmlformats.org/package/2006/relationships" =
xmlns:spwp =3D=20
"http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t =3D=20
"http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m =
=3D=20
"http://schemas.microsoft.com/exchange/services/2006/messages" =
xmlns:pptsl =3D=20
"http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl =
=3D=20
"http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksSe=
rvice"=20
XMLNS:Z =3D "urn:schemas-microsoft-com:" xmlns:st =3D "=01"><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<META content=3D"MSHTML 6.00.6000.16809" name=3DGENERATOR>
<STYLE>@font-face {
	font-family: Calibri;
}
@font-face {
	font-family: Tahoma;
}
@font-face {
	font-family: Consolas;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
	FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: =
"Calibri","sans-serif"
}
LI.MsoNormal {
	FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: =
"Calibri","sans-serif"
}
DIV.MsoNormal {
	FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: =
"Calibri","sans-serif"
}
A:link {
	COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
	COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
	COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
	COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P.MsoPlainText {
	FONT-SIZE: 10.5pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: Consolas; =
mso-style-priority: 99; mso-style-link: "Plain Text Char"
}
LI.MsoPlainText {
	FONT-SIZE: 10.5pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: Consolas; =
mso-style-priority: 99; mso-style-link: "Plain Text Char"
}
DIV.MsoPlainText {
	FONT-SIZE: 10.5pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: Consolas; =
mso-style-priority: 99; mso-style-link: "Plain Text Char"
}
P.MsoAcetate {
	FONT-SIZE: 8pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: =
"Tahoma","sans-serif"; mso-style-priority: 99; mso-style-link: "Balloon =
Text Char"
}
LI.MsoAcetate {
	FONT-SIZE: 8pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: =
"Tahoma","sans-serif"; mso-style-priority: 99; mso-style-link: "Balloon =
Text Char"
}
DIV.MsoAcetate {
	FONT-SIZE: 8pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: =
"Tahoma","sans-serif"; mso-style-priority: 99; mso-style-link: "Balloon =
Text Char"
}
SPAN.PlainTextChar {
	FONT-FAMILY: Consolas; mso-style-priority: 99; mso-style-link: "Plain =
Text"; mso-style-name: "Plain Text Char"
}
SPAN.BalloonTextChar {
	FONT-FAMILY: "Tahoma","sans-serif"; mso-style-priority: 99; =
mso-style-link: "Balloon Text"; mso-style-name: "Balloon Text Char"
}
SPAN.EmailStyle21 {
	COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: =
personal-reply
}
.MsoChpDefault {
	FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.Section1 {
	page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]--></HEAD>
<BODY lang=3DEN-US vLink=3Dpurple link=3Dblue>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D696364916-27042010><FONT =
face=3DArial=20
color=3D#0000ff size=3D2>I believe OSSEC, plus Splunk would be a great =
low/no cost=20
solution which would meet all of your requirements. Syslog-ng&nbsp;can =
also be=20
bolted on if you want something like file system-level log storage with=20
directories based on hostnames.</FONT></SPAN></DIV>
<DIV dir=3Dltr align=3Dleft><SPAN =
class=3D696364916-27042010></SPAN>&nbsp;</DIV>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D696364916-27042010><FONT =
face=3DArial=20
color=3D#0000ff size=3D2>I somewhat agree with the quote below if by =
"home grown"=20
solutions, Gartner means some internally developed scripts doing some=20
string/regex matching. If, on the other hand, they also include a =
collection of=20
mature open source tools to put the pieces together, I do not =
necessarily think=20
it's on the mark. Many of today's open-source log management/analysis =
solutions=20
rival the expensive commercial offerings and provide a lot of value for =
the=20
investment.</FONT></SPAN></DIV><BR>
<DIV class=3DOutlookMessageHeader lang=3Den-us dir=3Dltr align=3Dleft>
<HR tabIndex=3D-1>
<FONT face=3DTahoma size=3D2><B>From:</B> =
[email protected]=20
[mailto:[email protected]] <B>On Behalf Of =
</B>Youngquist,=20
Jason R.<BR><B>Sent:</B> Monday, April 26, 2010 10:04 AM<BR><B>To:</B>=20
'[email protected]'<BR><B>Subject:</B> [logs] Open Source =
centralized=20
log management/SIEM solutions<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=3DSection1>
<P class=3DMsoPlainText>Is anyone using any Open Source or low cost =
centralized=20
log management/SIEM solution in a production environment which you would =

recommend?<o:p></o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText>Specifically, I'm looking for:<o:p></o:p></P>
<P class=3DMsoPlainText>--scalability - must be able to handle hundreds =
of log=20
sources - majority being servers and network devices<o:p></o:p></P>
<P class=3DMsoPlainText>--good searching capability<o:p></o:p></P>
<P class=3DMsoPlainText>--ability to generate alerts<o:p></o:p></P>
<P class=3DMsoPlainText>--good reporting capability &#8211; pre-built =
reports would be=20
nice<o:p></o:p></P>
<P class=3DMsoPlainText>--a solution auditors would =
approve<o:p></o:p></P>
<P class=3DMsoPlainText>--able to meet regulatory requirements such as=20
PCI<o:p></o:p></P>
<P class=3DMsoPlainText>--fast implementation time &#8211; how long =
would it take to get=20
the solution up and running?<o:p></o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText>There are more things I&#8217;d like, but these =
are the big=20
requirements.<o:p></o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText>If an Open Source solution, are there any =
companies that=20
offer professional services (ie. consulting/configuration assistance) so =
we=20
could hit the ground running and not have to spend weeks/months=20
configuring/creating rules/reports, etc.&nbsp; Ideally, the solution =
should have=20
some commercial support behind it so if we run into any issues we can =
speak to a=20
knowledgeable person.<o:p></o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText>For those QSAs out there, are there any Open =
Source=20
solutions/low-cost solutions that you have seen implemented well and =
meet the=20
PCI regulatory guidelines?&nbsp; If so, what were they?&nbsp; If not, =
what were=20
they lacking that commercial products provide?<o:p></o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText>For those of you with a home-grown/Open Source =
log=20
management solution, do you agree with the Gartner quote below?&nbsp; =
Why/why=20
not?&nbsp; <o:p></o:p></P>
<P class=3DMsoPlainText style=3D"MARGIN-LEFT: 0.5in">According to =
Gartner=20
researchers, "Although [home-grown log management] may prove effective =
for a=20
limited set of data sources with clearly defined "strings" that the =
organization=20
is searching for, most organizations quickly run into scalability =
issues, as=20
well as issues using the data for situational awareness in support of =
incident=20
response. In most cases, internally developed centralized application =
log=20
solutions will fall short of meeting organizational=20
requirements."<o:p></o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText>If you had to do it again would you &#8220;roll =
your own=20
solution&#8221; or purchase a commercial log management product? =
<o:p></o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText>Appreciate any information you can =
provide.<o:p></o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P>
<P class=3DMsoPlainText>Thanks.<o:p></o:p></P>
<P class=3DMsoPlainText>Jason Youngquist<o:p></o:p></P>
<P class=3DMsoPlainText>Information Technology Security Engineer,=20
Security+<o:p></o:p></P>
<P class=3DMsoPlainText>Technology Services<o:p></o:p></P>
<P class=3DMsoPlainText>Columbia College<o:p></o:p></P>
<P class=3DMsoPlainText>1001 Rogers Street, Columbia, MO&nbsp;=20
65216<o:p></o:p></P>
<P class=3DMsoPlainText>(573) 875-7334<o:p></o:p></P>
<P class=3DMsoPlainText>[email protected]<o:p></o:p></P>
<P class=3DMsoPlainText>http://www.ccis.edu<o:p></o:p></P>
<P class=3DMsoPlainText>&nbsp;<o:p></o:p></P>
<P class=3DMsoPlainText><o:p>&nbsp;</o:p></P></DIV></BODY></HTML>

------_=_NextPart_001_01CAE62A.847244B0--

--===============1802345070==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
LogAnalysis mailing list
[email protected]
http://www.loganalysis.org/mailman/listinfo/loganalysis
--===============1802345070==--