Having problems getting thresholding to work with snmp logmatch

JohnD Blackburn <[email protected]> Thu, 22 Nov 2018 03:24:11 +0000
Newsgroups gmane.network.opennms.general
Message-ID <AM6PR0602MB3381262D31E1D7B08E9FC3C2CBDB0@AM6PR0602MB3381.eurprd06.prod.outlook.com>
--===============8226636507307028484==
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_AM6PR0602MB3381262D31E1D7B08E9FC3C2CBDB0AM6PR0602MB3381_"

--_000_AM6PR0602MB3381262D31E1D7B08E9FC3C2CBDB0AM6PR0602MB3381_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi All

I've been trying to set up a notification using the snmp logmatch feature w=
hich I want to raise a notification if a specific message appears in an app=
lication log

I have the following code set up in snmpd.conf on the client:

   logmatch batchFailure /appliance/data/efs/dev/batch/batch.log 600 c.m.b.=
n.NotificationPollingService - Unable to read/process new notification requ=
ests

and in collectd-configuration I have:

   <package name=3D"ABBCS-logmatch" remote=3D"false">
      <filter>(IPADDR !=3D '0.0.0.0') &amp; (catincRFIntegration | catincBa=
tchFailures )</filter>
      <include-range begin=3D"1.1.1.1" end=3D"254.254.254.254"/>
      <include-range begin=3D"::1" end=3D"ffff:ffff:ffff:ffff:ffff:ffff:fff=
f:ffff"/>
      <service name=3D"SNMP" interval=3D"300000" user-defined=3D"false" sta=
tus=3D"on">
         <parameter key=3D"collection" value=3D"ABBCS-logmatch"/>
         <parameter key=3D"thresholding-enabled" value=3D"true"/>
         <parameter key=3D"port" value=3D"161"/>
         <parameter key=3D"retry" value=3D"1"/>
         <parameter key=3D"timeout" value=3D"30000"/>
         <parameter key=3D"oid" value=3D".1.3.6.1.4.1.2021.16.2.1.1.1"/>
      </service>
   </package>

In datacollection/ABBCS-logmatch.xml, I have the following:
<datacollection-group name=3D"ABBCS-logmatch">
    <group name=3D"ABBCS-logmatch-batch" ifType=3D"ignore">
        <mibObj oid=3D".1.3.6.1.4.1.2021.16.2.1.5" instance=3D"1" alias=3D"=
batchfailure" type=3D"Counter32" />
    </group>
    <systemDef name=3D"ABBCS-batch">
        <sysoidMask>.1.3.6.1.4.1.8072.3.2.10</sysoidMask>
        <collect>
           <includeGroup>ABBCS-logmatch-batch</includeGroup>
        </collect>
    </systemDef>
</datacollection-group>


Threshd-configuration.xml:
   <package name=3D"ABBCS-logmatch">
      <filter>IPADDR !=3D '0.0.0.0'</filter>
      <include-range begin=3D"1.1.1.1" end=3D"254.254.254.254"/>
      <include-range begin=3D"::1" end=3D"ffff:ffff:ffff:ffff:ffff:ffff:fff=
f:ffff"/>
      <service name=3D"ABBCS-logmatch" interval=3D"300000" user-defined=3D"=
false" status=3D"on">
         <parameter key=3D"thresholding-group" value=3D"ABBCS-logmatch"/>
      </service>
   </package>

Thresholds.xml:
   <group name=3D"ABBCS-logmatch" rrdRepository=3D"/opt/opennms/share/rrd/s=
nmp/">
      <threshold description=3D"Trigger an alert for batch failures" type=
=3D"high" ds-type=3D"node" value=3D"1" rearm=3D"0.9" trigger=3D"1" ds-label=
=3D"batchfailure" triggeredUEI=3D"uei.opennms.org/ABBCS/Batch/failure-crit"=
 rearmedUEI=3D"uei.opennms.org/ABBCS/Batch/failure-rearm" filterOperator=3D=
"OR" ds-name=3D"batchfailure"/>
   </group>


I know OpenNMS is successfully fetching the data from snmp because of this =
entry in the collectd.log:

2018-11-22 08:02:42,359 INFO  [Collectd-Thread-29-of-50] o.o.n.c.p.r.RrdPer=
sistOperationBuilder: updateRRD: updating RRD file /opt/opennms/share/rrd/s=
nmp/fs/Servers/ellbat0-dev-n03-kum/batchfailure.jrb with values '1542852162=
:8'

But I'm getting the following error in my collectd.log:

2018-11-22 08:02:42,360 DEBUG [Collectd-Thread-29-of-50] o.o.n.t.Thresholdi=
ngSet: applyThresholds: Ignoring resource node[537].interfaceSnmp[eth0-000d=
3a3992e3] because required attributes map is empty.


What am I missing to get this threshold to work?

Regards,
John




--_000_AM6PR0602MB3381262D31E1D7B08E9FC3C2CBDB0AM6PR0602MB3381_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Verdana;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
	{font-family:ABBvoice;
	panose-1:2 13 6 3 2 5 3 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Verdana",sans-serif;
	color:windowtext;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-AU" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">Hi All<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">I&#8217;ve been trying to set up a notification using the snmp logma=
tch feature which I want to raise a notification if a specific message appe=
ars in an application log<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">I have the following code set up in snmpd.conf on the client:<o:p></=
o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp; logmatch batchFailure /appliance/data/efs/dev/batch/batch.log =
600 c.m.b.n.NotificationPollingService - Unable to read/process new notific=
ation requests<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">and in collectd-configuration I have:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp; &lt;package name=3D&quot;ABBCS-logmatch&quot; remote=3D&quot;f=
alse&quot;&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;filter&gt;(IPADDR !=3D '0.0.0.0') &amp;a=
mp; (catincRFIntegration | catincBatchFailures )&lt;/filter&gt;<o:p></o:p><=
/span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;include-range begin=3D&quot;1.1.1.1&quot=
; end=3D&quot;254.254.254.254&quot;/&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;include-range begin=3D&quot;::1&quot; en=
d=3D&quot;ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff&quot;/&gt;<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;service name=3D&quot;SNMP&quot; interval=
=3D&quot;300000&quot; user-defined=3D&quot;false&quot; status=3D&quot;on&qu=
ot;&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;parameter key=3D&quot;=
collection&quot; value=3D&quot;ABBCS-logmatch&quot;/&gt;<o:p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;parameter key=3D&quot;=
thresholding-enabled&quot; value=3D&quot;true&quot;/&gt;<o:p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;parameter key=3D&quot;=
port&quot; value=3D&quot;161&quot;/&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;parameter key=3D&quot;=
retry&quot; value=3D&quot;1&quot;/&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;parameter key=3D&quot;=
timeout&quot; value=3D&quot;30000&quot;/&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;parameter key=3D&quot;=
oid&quot; value=3D&quot;.1.3.6.1.4.1.2021.16.2.1.1.1&quot;/&gt;<o:p></o:p><=
/span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;/service&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp; &lt;/package&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">In datacollection/ABBCS-logmatch.xml, I have the following:<o:p></o:=
p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&lt;datacollection-group name=3D&quot;ABBCS-logmatch&quot;&gt;<o:p></o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp; &lt;group name=3D&quot;ABBCS-logmatch-batch&quot; ifType=
=3D&quot;ignore&quot;&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;mibObj oid=3D&quot;.1.3.6.1.=
4.1.2021.16.2.1.5&quot; instance=3D&quot;1&quot; alias=3D&quot;batchfailure=
&quot; type=3D&quot;Counter32&quot; /&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp; &lt;/group&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp; &lt;systemDef name=3D&quot;ABBCS-batch&quot;&gt;<o:p></o=
:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;sysoidMask&gt;.1.3.6.1.4.1.8=
072.3.2.10&lt;/sysoidMask&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;collect&gt;<o:p></o:p></span=
></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;includeGro=
up&gt;ABBCS-logmatch-batch&lt;/includeGroup&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;/collect&gt;<o:p></o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp; &lt;/systemDef&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&lt;/datacollection-group&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">Threshd-configuration.xml:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp; &lt;package name=3D&quot;ABBCS-logmatch&quot;&gt;<o:p></o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;filter&gt;IPADDR !=3D '0.0.0.0'&lt;/filt=
er&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;include-range begin=3D&quot;1.1.1.1&quot=
; end=3D&quot;254.254.254.254&quot;/&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;include-range begin=3D&quot;::1&quot; en=
d=3D&quot;ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff&quot;/&gt;<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;service name=3D&quot;ABBCS-logmatch&quot=
; interval=3D&quot;300000&quot; user-defined=3D&quot;false&quot; status=3D&=
quot;on&quot;&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;parameter key=3D&quot;=
thresholding-group&quot; value=3D&quot;ABBCS-logmatch&quot;/&gt;<o:p></o:p>=
</span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;/service&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp; &lt;/package&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">Thresholds.xml:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp; &lt;group name=3D&quot;ABBCS-logmatch&quot; rrdRepository=3D&q=
uot;/opt/opennms/share/rrd/snmp/&quot;&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;threshold description=3D&quot;Trigger an=
 alert for batch failures&quot; type=3D&quot;high&quot; ds-type=3D&quot;nod=
e&quot; value=3D&quot;1&quot; rearm=3D&quot;0.9&quot; trigger=3D&quot;1&quo=
t; ds-label=3D&quot;batchfailure&quot; triggeredUEI=3D&quot;uei.opennms.org=
/ABBCS/Batch/failure-crit&quot;
 rearmedUEI=3D&quot;uei.opennms.org/ABBCS/Batch/failure-rearm&quot; filterO=
perator=3D&quot;OR&quot; ds-name=3D&quot;batchfailure&quot;/&gt;<o:p></o:p>=
</span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
&nbsp;&nbsp; &lt;/group&gt;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">I know OpenNMS is successfully fetching the data from snmp because o=
f this entry in the collectd.log:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
2018-11-22 08:02:42,359 INFO&nbsp; [Collectd-Thread-29-of-50] o.o.n.c.p.r.R=
rdPersistOperationBuilder: updateRRD: updating RRD file /opt/opennms/share/=
rrd/snmp/fs/Servers/ellbat0-dev-n03-kum/batchfailure.jrb
 with values '1542852162:8'<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">But I&#8217;m getting the following error in my collectd.log:<o:p></=
o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Courier New&quot;">=
2018-11-22 08:02:42,360 DEBUG [Collectd-Thread-29-of-50] o.o.n.t.Thresholdi=
ngSet: applyThresholds: Ignoring resource node[537].interfaceSnmp[eth0-000d=
3a3992e3] because required attributes map is empty.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">What am I missing to get this threshold to work?<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">Regards,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:&quot;Verdana&quot;,sans-=
serif">John<o:p></o:p></span></p>
<p class=3D"MsoNormalCxSpMiddle" style=3D"mso-margin-bottom-alt:auto;mso-ad=
d-space:auto">
<span style=3D"font-size:9.0pt;font-family:&quot;ABBvoice&quot;,sans-serif;=
mso-fareast-language:EN-AU"><o:p>&nbsp;</o:p></span></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
</body>
</html>

--_000_AM6PR0602MB3381262D31E1D7B08E9FC3C2CBDB0AM6PR0602MB3381_--


--===============8226636507307028484==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline


--===============8226636507307028484==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
--===============8226636507307028484==--