Re: OpenNMS monitoring and data collection from the Azure REST API

With the XML and JSON collection I'm doing, I create a detector to identify
the specific device or system type via some other means.  For example, in
one case I am using the WebDetector to identify the class of device based
on regex match in the output, and the service it creates
has data collection defined via the XmlCollector.  I'm not sure if that's a
tactic that would be helpful to you, as I don't know what Azure VMs "look
like" from the outside of what identifying information you could use.

D

On Mon, May 7, 2018 at 10:21 PM, JohnD Blackburn <[email protected]
> wrote:

> Thanks for that Dino, I’ve read that document already.  However, it
> doesn’t help answer my question about the fact that the authentication
> token needed to be passed in every Azure REST API call as an HTTP header
> is only valid for an hour and a separate API call is required to get a new
> access token, So I’m trying to figure out the best way to get OpenNMS to
> have the correct URI.  The doco is also not clear how to map the JSON data
> to fields in the opennms XML config file.  The JSON data returned from
> Azure API is sufficiently different to the documented example that I’m
> getting confused as to how I should be referencing it.  But that could be
> just that I was looking at it at the end of the week, and/or needed more
> coffee :D
>
>
>
>
>
>
>
>
>
> I think I’ve managed to get something working by using fields in the asset
> info which I’ve got a shell script that updates the password field in the
> Authentication section with the current access token.
>
>
>
> xml-datacollection-config.xml contains:
>
>
>
>     <xml-collection name="xml-Azure-test">
>
>         <rrd step="300">
>
>             <rra>RRA:AVERAGE:0.5:1:2016</rra>
>
>             <rra>RRA:AVERAGE:0.5:12:1488</rra>
>
>             <rra>RRA:AVERAGE:0.5:288:366</rra>
>
>             <rra>RRA:MAX:0.5:288:366</rra>
>
>             <rra>RRA:MIN:0.5:288:366</rra>
>
>         </rrd>
>
>         <xml-source url="https://management.azure.
> com/{assetnumber}/providers/Microsoft.ResourceHealth/
> availabilityStatuses/current">
>
>             <request method="GET">
>
>                 <parameter name="api-version" value="2017-07-01"/>
>
>                 <parameter name="$expand" value="recommendations"/>
>
>                 <header name="Authorization" value="Bearer {password}"/>
>
>                 <header name="Content-Type" value="application/json"/>
>
>             </request>
>
>             <import-groups>xml-datacollection/ABBCS-azure-
> test.xml</import-groups>
>
>         </xml-source>
>
>     </xml-collection>
>
>
>
> However, trying to set up a detector in Default Foreign Source, I’m having
> trouble specifying the http headers required to implement this in the
> detector:
>
>
>
>         <xml-source url="https://management.azure.
> com/{assetnumber}/providers/Microsoft.ResourceHealth/
> availabilityStatuses/current">
>
>             <request method="GET">
>
>                 <parameter name="api-version" value="2017-07-01"/>
>
>                 <parameter name="$expand" value="recommendations"/>
>
>                 <header name="Authorization" value="Bearer {password}"/>
>
>                 <header name="Content-Type" value="application/json"/>
>
>             </request>
>
>             <import-groups>xml-datacollection/ABBCS-azure-
> test.xml</import-groups>
>
>         </xml-source>
>
>
>
> Without passing that authorization header in the get request, I cannot get
> the correct response code to validate “detection”
>
>
>
> Any ideas on how to detect this?
>
> (note: the assetnumber field comes from the Asset Info record in OpenNMS
> and is the field I picked to store the Azure ResourceID, as there isn’t
> really any Azure(/generic cloud provider) specific fields in the Asset Info
> record, so I just picked one…)
>
>
>
> Or is using a detector in this instance the wrong way to go about this?
>
>
>
> Regards
>
> John.
>
>
>
>
>
>
>
>
>
>
>
> *From:* Dino Yancey [mailto:[email protected]]
> *Sent:* Friday, 4 May 2018 10:16 PM
> *To:* General OpenNMS Discussion <[email protected]>
> *Subject:* Re: [opennms-discuss] OpenNMS monitoring and data collection
> from the Azure REST API
>
>
>
>
>
> https://wiki.opennms.org/wiki/XML_Collector#JSON_Collector
>
>
>
> On Thu, May 3, 2018 at 10:09 PM, JohnD Blackburn <
> [email protected]> wrote:
>
> Hi All,
>
>
>
> We are currently using OpenNMS 21.1.0 to monitor all our Azure VM’s, and
> it has been brought up that there is a whole bunch of metrics and statuses
> that are only available through Azure.  I have been looking at options for
> getting some of this info into OpenNMS, and have found several potential
> methods for achieving this.
>
>
>
> It is possible to set up webhooks in Azure that get called when events,
> statuses, and thresholds are triggered, which could possibly work, but
> would require me to set up an internet facing webserver to receive the
> webhook calls, and to then process the received JSON data to the correct
> node in OpenNMS.  This would also require configuration added to Azure to
> “send” these details.
>
>
>
> A second option I have come across is to use the Azure REST API which is
> documented here: https://docs.microsoft.com/en-us/rest/api/azure/
>
>
>
> Using this should allow me to drive the collection from the OpenNMS
> server, however I would need to get the mapping between OpenNMS nodes and
> Azure resources working.
>
>
>
> To be able to get details out of Azure for a specific node, I would need
> to know the “SubscriptionID”, “ResourceGroupName” and “ResourceID” from
> Azure, and to complicate matters, requests to the Azure API require an
> access token which I can get with the following curl commandline which
> returns the access token in a JSON formatted file
>
>
>
>     curl -X POST -d 'grant_type=client_credentials&client_id='$
> CLIENTID'&client_secret='$CLIENTSECRET'&resource=https%3A%2F%
> 2Fmanagement.azure.com%2F' https://login.microsoftonline.
> com/$TENANTID/oauth2/token > $TOKEN_FILE
>
>
>
> This Access token that is returned expires after an hour, so a new access
> token would need to be requested every hour.   The access token is passed
> into the HTTP request as a header.  The following curl command shows the
> format required for the http request:
>
>
>
>    curl -X GET -H "Authorization: Bearer $TOKEN_ID" -H "Content-Type:
> application/json" https://management.azure.com/$URI
>
>
>
>
>
> That access token is 1335 characters long.
>
>
>
> What is the best way to capture data from the Azure REST API?  Do I need a
> script that runs hourly to update the access token in the asset comment
> field and store all those variables in Asset Fields?  Would a 1335
> character string be storable in an asset field?
>
>
>
> Has anyone implemented any OpenNMS monitoring/data collection from the
> Azure REST API?
>
>
>
>
>
>
>
> Also, Trying to create a collector for the JSON, given the following JSON
> as returned by Azure:
>
> {
>
>   "id": "/subscriptions/SubscriptionID/resourceGroups/
> ResourceGroupName/providers/Microsoft.Compute/virtualMachines/VMName/
> providers/Microsoft.ResourceHealth/availabilityStatuses/current",
>
>   "name": "current",
>
>   "type": "Microsoft.ResourceHealth/AvailabilityStatuses",
>
>   "location": "australiaeast",
>
>   "properties": {
>
>     "availabilityState": "Available",
>
>     "summary": "There aren't any known Azure platform problems affecting
> this virtual machine",
>
>     "detailedStatus": "",
>
>     "reasonType": "",
>
>     "occuredTime": "2018-04-26T09:15:52Z",
>
>     "reasonChronicity": "Persistent",
>
>     "reportedTime": "2018-05-04T02:02:47.222185Z"
>
>   }
>
> }
>
>
>
> How would I format the XML Group definition in xml-datacollection?  It is
> not clear to me from the documentation.
>
> I’m guessing something like this:
>
> <xml-groups>
>
>     <xml-group name="azure-test" resource-type="node"
>
>                 resource-xpath="/"
>
>                 key-xpath="@id">
>
>       <xml-object name="name" type=”string" xpath="@name" />
>
>       <xml-object name="type" type="string" xpath="@type" />
>
>       <xml-object name="location" type="string" xpath="@location" />
>
>       <xml-object name="availabilityState" type="string"
> xpath="properties[@availabilityState]" />
>
>       <xml-object name="summary" type="string"
> xpath="properties[@summary]" />
>
>       <xml-object name="detailedStatus" type="string" xpath="properties[@detailedStatus]"
> />
>
>       <xml-object name="reasonType" type="string"
> xpath="properties[@reasonType]" />
>
>       <xml-object name="occuredTime" type="string" xpath="properties[@occuredTime]"
> />
>
>       <xml-object name="reasonChronicity" type="string" xpath="properties[@reasonChronicity]"
> />
>
>       <xml-object name="reportedTime" type="string" xpath="properties[@reportedTime]"
> />
>
>     </xml-group>
>
> </xml-groups>
>
>
>
>
>
> Regards,
>
> John Blackburn
>
>
>
>
>
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of
> this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of
> this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss