Re: Parsing 100M of MIBs

Just a quick note that this tool [1] should be able to dump MIBs into individual [2] JSON files and maps of MIB objects [3].

May be you could then load such JSONs into a DB and make your service talking to that DB to perform the translation you need?

Not sure how relevant my suggestion would be to your situation. Noting this just in case. ;-)

1. https://github.com/etingof/pysmi
2. http://mibs.snmplabs.com/json/fulltexts/A3COM-HUAWEI-3GMODEM-MIB.json
3. http://mibs.snmplabs.com/json/index.json

> On 25 Jan 2017, at 22:25, Chris Fowler <[email protected]> wrote:
> 
> I'm tying to do something that works on a smaller scale, but on a
> larger scale keeping up with MIB syntax errors and such is a problem.
> 
> I have a device that has 25M of MIB files.  snmptranslate takes too
> long to process through '+ALL'. I've used the SNMP Perl module so I
> wrote a translation service using SOAP for my applications.
> I like it.  It uses SNMP::initMib() to go after the 25 and I use
> smidump, etc to get extra data for the API.  Things like
> 'LAST-UPDATED'.  It also parses traps out and stores that data for
> callers.
> 
> As I receive zip files with vendor MIBs I add this to the repository,
> but I want to cache 200M worth of data on another system.  I'll modify
> my service on the device to cache when OIDs are seen from traps, but
> it can not translate.  I've ported the service over to the server.
> 
> Here is where I've ran into problems:
> 
> I've downloaded a few repositories of MIBS off github.  LibreNMS is
> one such repository.
> 
> Net-SNMP MIBs and the RFC oness generated in Net-SNMP build are stored
> in /usr/share/snmp/mibs.
> 
> The ones I've added to the device are in
> /usr/share/snmp/private
> 
> The large group is in
> /usr/share/snmp/extra
> 
> MIBDIRS on those and MIBS=+ALL will be successful for the first 2, but
> the last one has many errors.   My line of thinking is that if
> LibreNMS uses those are they running into the same thing?  If not,
> maybe I'm doing something wrong.
> 
> Management is an issue that I also need to tackle. "Techs" that "want
> to monitor" will provide zips of files they do not know.  "These are
> the MIBs".  I may have those already.  I used smidump to get the last
> revision date so I could maintain the newest versions of these files.
> I use a script to rename the files to the name of the MIB MODULE
> contained within.  To copy a MIB to one of those directories it must
> be pre-processed.
> 
> The service works.  I like it and it seems to work well.  I know that
> creating a translation service is not impossible.   I don't understand
> MIBs within this system enough to know if using a service for
> translation is really possible.  Perl's SNMP with +ALL does not
> provide a way to sort some of this out so I'm doing it as prep work.
> 
> I've tried a few tricks to stop using snmptranslate when parsing
> traps.  I want the labels in log files for the techs (not OIDs).  The
> device that has 25M will consume 7-9s of sys time before even
> receiving a response from snmptranslate. If I'm getting traps every 5s
> I'll never catch up.  Using the service it is < 1s to get an answer.
> Only one &SNMP::initMib() has to be executed at boot vs one for each
> trap!
> 
> I've tried a walk through $SNMP::MIB and storing the values in a db
> file.  The result is 19M, large, and the reality is the the MIBs are
> the database.  I've tried caching the results of a translate in a DB
> file.  This works great until you run into .N where N may be an
> interface, etc and can be 1-1000.  The DB file and the caching of past
> translations in a db file did not scale well.  I always came back to
> $SNMP::MIB.  I thought maybe I could Storable::n_freeze(SNMP::MIB) to
> disk and use that instead.  The only time the MIB files change is when
> I regenerate a new image on the device.  I'm sure that hash is a tied
> hash to the Net-SNMP C API and can't be stored.
> 
> I'd love to drop my devices down to just the Net-SNMP MIBs plus RFC,
> program their service to send OIDs upstream for translation and cache
> those locally till next boot.  I can write that.  I'm not sure if it
> is really possible with all these MIBs I collect from all over.
> 
> Ideas and suggestions on this issue would be greatly appreciated,
> 
> Chris
> 
> -----------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Net-snmp-users mailing list
[email protected]
Please see the following page to unsubscribe or change other options:
https://lists.sourceforge.net/lists/listinfo/net-snmp-users