Re: Netsaint-users digest, Vol 1 #1432 - 2 msgs
amol sapkal <[email protected]>
| Newsgroups | gmane.network.netsaint.user |
|---|---|
| Message-ID | <[email protected]> |
Hi Roger, Thx a ton! It did work! Thx to Stanley too, for helping me out! I had already written a auto edition code in C for this, but it was giving me some errors. although its not that great, but would like to contribute it to the netsaint-users group, when ready and tested. I hope thats allowed. Thx again to both of you. Regds, Amol Sapkal GTL Ltd. [email protected] wrote:Send Netsaint-users mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit https://lists.sourceforge.net/lists/listinfo/netsaint-users or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of Netsaint-users digest..." Today's Topics: 1. Re: System time problem with NetSaint (Roger Flemming/SYD/CEtv) 2. nrpe-1.5 on AIX 4.3.3 (Tyson Harker) --__--__-- Message: 1 To: [email protected] Cc: amol sapkal Subject: Re: [netsaint] System time problem with NetSaint From: "Roger Flemming/SYD/CEtv" Date: Mon, 12 Aug 2002 18:07:27 +1100 This is a multipart message in MIME format. --=_alternative 0027337FCA256C13_= Content-Type: text/plain; charset="us-ascii" Hello Amol, I am sure many people will answer your question but I haven't seen any yet, so here goes. Also I hope the list doesn't mind that this post is basically all Perl, very little Netsaint! > Dear Sir, > Thx for the solution! > But im unable to decipher the command here.. > perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log It's a Perl one-liner script. -i.bak says edit in place, backing up the original with extension .bak. -e says the first argument is the actual script to execute, instead of the script's name. -p says take in a file (the second argument) line by line, putting each line into the default argument $_, execute the script, and printout $_. Thus the file (in this case netsaint.log) will have each line transformed by the contents of that script, with the original saved as netsaint.log.bak. The script itself use the pattern-match-and-substitute operator s/// to search for the start of a line (^) immediately followed by a literal left bracket (\[) followed by one or more digits (\d+) and a right bracket (\]). This is then replaced by $1 - Your_Time_Correction(eg 2*3600), where $1 is anything matched inside the parenthesis earlier (i.e. the digits) and the rest was meant to be replaced by you with your actual correction factor. That is, if the error was 2 hours 3 seconds, you'd actually put #$1 - 7203#. The 'e' right at the end says the replacement string is to be evaluated as an expression, i.e. replace the matched pattern with the digits minus the correction factor. Note there may be a small error here: while the date gets corrected, the square brackets will vanish. It might be better to put perl -i.bak -pe 's#^\[(\d+)\]#"[".($1 - 7200)."]"#e' netsaint.log which should keep the square brackets > is it necessary that it has to be in multiples of 3600 (1 hr) No. The time argument is simply seconds. By 'eg 2*3600' he meant put 7200 if you want 2 hours. > also pls tell me where this command has to be executed. If Perl has been set up correctly on your system you should be able to execute it from anywhere. Note that if you are not in the directory where netsaint.log lives, you will need to give its full path. > Also, I was unable to find any relations between the 10 digit time stamps (eg. 10288....), and the corresponding time value. Please help me to know the logic behind the generation of these time stamps. I dont find any common link between the sequence of the time stamps. Is it using some knind of algorithm? These time stamps are a standard Unix internal way of representing time, commonly known as t_time or epoch time. They represent the number of non-leap seconds past midnight, 1st January 1970, UTC (usually as a signed 32 bit integer) which is roughly how old Unix is. Yes this does mean you will need to get a 64 bit machine before the 2038 crisis! This date format has many advantages: - it is constant through out the world, i.e. time zone independent; - it is very compact (one 32 bit int for seconds, minutes, hours, days, months, and years); - apart from the word size problem in 2038, it runs forever into the past and future without any funny glitches or historical weirdness; - it is very easy to do arithmetic in it, while most other formats are horribly confusing; and - it is very portable. On the down side, it is hard for humans to read, but while we're talking Perl, you can convert epoch dates to human readable format in your own timezone, with: perl -e 'print scalar localtime 1029132827' (Substituting the number in question for the one I entered.) Similarly if you want the current epoch date just go: perl -e 'print time' [...] Stanley wrote: > perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log > > may be something like what you want, provided you know that the log > entries are __all__ are out by the same constant amount.[...] Or if there is a time before/after which they are bad by a constant amount, you could try something like: perl -i.bak -pe 's#^\[(\d+)\]#$1<1028132827 ? $1 - 7200 : $1#e' netsaint.log Substituting the epoch date where it all went wrong for 1028132827 and the actual error for 7200. May still be some hand editing at the cross-over point if the bad dates went forward in time, though. Cheers, Roger *************************************************************************************************************************************************************************** This email and any files transmitted with it, are confidential and is intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This footnote also confirms that this email message has been scanned by AUSTAR Communications content and virus scanning applications for the presence of computer viruses. *************************************************************************************************************************************************************************** --=_alternative 0027337FCA256C13_= Content-Type: text/html; charset="us-ascii" Hello Amol, I am sure many people will answer your question but I haven't seen any yet, so here goes. Also I hope the list doesn't mind that this post is basically all Perl, very little Netsaint! > Dear Sir, > Thx for the solution! > But im unable to decipher the command here.. > perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log It's a Perl one-liner script. -i.bak says edit in place, backing up the original with extension .bak. -e says the first argument is the actual script to execute, instead of the script's name. -p says take in a file (the second argument) line by line, putting each line into the default argument $_, execute the script, and printout $_. Thus the file (in this case netsaint.log) will have each line transformed by the contents of that script, with the original saved as netsaint.log.bak. The script itself use the pattern-match-and-substitute operator s/// to search for the start of a line (^) immediately followed by a literal left bracket (\[) followed by one or more digits (\d+) and a right bracket (\]). This is then replaced by $1 - Your_Time_Correction(eg 2*3600), where $1 is anything matched inside the parenthesis earlier (i.e. the digits) and the rest was meant to be replaced by you with your actual correction factor. That is, if the error was 2 hours 3 seconds, you'd actually put #$1 - 7203#. The 'e' right at the end says the replacement string is to be evaluated as an expres sion, i.e. replace the matched pattern with the digits minus the correction factor. Note there may be a small error here: while the date gets corrected, the square brackets will vanish. It might be better to put perl -i.bak -pe 's#^\[(\d+)\]#"[".($1 - 7200)."]"#e' netsaint.log which should keep the square brackets > is it necessary that it has to be in multiples of 3600 (1 hr) No. The time argument is simply seconds. By 'eg 2*3600' he meant put 7200 if you want 2 hours. > also pls tell me where this command has to be executed. If Perl has been set up correctly on your system you should be able to execute it from anywhere. Note that if you are not in the directory where netsaint.log lives, you will need to give its full path. > Also, I was unable to find any relations between the 10 digit time stamps (eg. 10288....), and the corresponding time value. Please help me to know the logic behind the generation of these time stamps. I dont find any common link between the sequence of the time stamps. Is it using some knind of algorithm? These time stamps are a standard Unix internal way of representing time, commonly known as t_time or epoch time. They represent the number of non-leap seconds past midnight, 1st January 1970, UTC (usually as a signed 32 bit integer) which is roughly how old Unix is. Yes this does mean you will need to get a 64 bit machine before the 2038 crisis! This date format has many advantages: - it is constant through out the world, i.e. time zone independent; - it is very compact (one 32 bit int for seconds, minutes, hours, days, months, and years); - apart from the word size problem in 2038, it runs forever into the past and future without any funny glitches or historical weirdness; - it is very easy to do arithmetic in it, while most other formats are horribly confusing; and - it is very portable. On the down side, it is hard for humans to read, but while we're talking Perl, you can convert epoch dates to human readable format in your own timezone, with: perl -e 'print scalar localtime 1029132827' (Substituting the number in question for the one I entered.) Similarly if you want the current epoch date just go: perl -e 'print time' [...] Stanley wrote: > perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log > > may be something like what you want, provided you know that the log > entries are __all__ are out by the same constant amount.[...] Or if there is a time before/after which they are bad by a constant amount, you could try something like: perl -i.bak -pe 's#^\[(\d+)\]#$1<1028132827 ? $1 - 7200 : $1#e' netsaint.log Substituting the epoch date where it all went wrong for 1028132827 and the actual error for 7200. May still be some hand editing at the cross-over point if the bad dates went forward in time, though. Cheers, Roger *************************************************************************************************************************************************************************** This email and any files transmitted with it, are confidential and is intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This footnote also confirms that this email message has been scanned by AUSTAR Communications content and virus scanning applications for the presence of computer viruses. *************************************************************************************************************************************************************************** --=_alternative 0027337FCA256C13_=-- --__--__-- Message: 2 From: Tyson Harker To: "'[email protected]'" Date: Mon, 12 Aug 2002 08:49:03 -0700 Subject: [netsaint] nrpe-1.5 on AIX 4.3.3 I had everything running great and with no problems, then I found out that the nrpe daemon wouldnt allow me to "down" the software that my company uses so I decided to go in and play with the permissions that nrpe runs under. After getting no luck, I decided to put everything back as it was, but when I go to run the daemon useing ./nrpe -d /etc/nrpe.cfg it never runs. I go a grep for the process and it never shows up. So I wiped out every instance of nrpe that I could find then reconfigured it and I still get the same problem. Does anyone have any idea why I suddenly cannot run this? -----Original Message----- From: [email protected] [mailto:[email protected]] Sent: Saturday, August 10, 2002 12:03 PM To: [email protected] Subject: Netsaint-users digest, Vol 1 #1431 - 3 msgs Send Netsaint-users mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit https://lists.sourceforge.net/lists/listinfo/netsaint-users or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of Netsaint-users digest..." Today's Topics: 1. Re: System time problem with NetSaint (amol sapkal) 2. Detecting anomalies in time series. Was: checking for glacial changes (Stanley Hopcroft) 3. Re: Detecting anomalies in time series. Was: checking for glacial changes (Stanley Hopcroft) -- __--__-- Message: 1 Date: Fri, 9 Aug 2002 15:45:38 -0700 (PDT) From: amol sapkal Subject: Re: [netsaint] System time problem with NetSaint To: Stanley Hopcroft , [email protected] Cc: [email protected] --0-1857066914-1028933138=:31770 Content-Type: text/plain; charset=us-ascii Dear Sir, Thx for the solution! But im unable to decipher the command here.. perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log is it necessary that it has to be in multiples of 3600 (1 hr) also pls tell me where this command has to be executed. Also, I was unable to find any relations between the 10 digit time stamps (eg. 10288....), and the corresponding time value. Please help me to know the logic behind the generation of these time stamps. I dont find any common link between the sequence of the time stamps. Is it using some knind of algorithm? Will manual edition help, once the relation is known? PS : I got 2 log files automatically generated for that day (8 august), one with the name.. netsaint-08-08-2002-00.log and other with netsaint-08-08-2002-18.log The system date change was approximately 18-19 hrs ahead of the original time. Is the value 18 in the 2nd log file to do anything with it? your kind help is appreciated. Warm Regds, Amol Sapkal GTL Ltd., Mumbai Stanley Hopcroft wrote:Dear Sir, On Thu, Aug 08, 2002 at 02:20:52PM -0700, amol sapkal wrote: > > Hi All, > > I'm facing a strange but yet significant problem, where in my machine's system timing was changed unknowingly. The NetSaint logs now show me wrong log entries. > > Is there anyway to get back the correct entries? > I thhink that the only way of getting them back is editing them with the swiss army chainsaw, making sure that you are careful not to take your legs off. > Do I need to edit the netsaint logs manually. > perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log may be something like what you want, provided you know that the log entries are __all__ are out by the same constant amount. (in any case, you should try your auto edit program [if you choose to use one] on a copy of the log. -i.bak will give you the original log with a '.bak' suffix/extension). Otherwise, you'll have to identify when the log got the bad timestamps and then fix that bit. > I have restored the correct system timing now. It really sounds like you must . identify the bad bits in the log . cut them out and save them somewhere . correct them . put the corrected bits back in This is feasable (I have done something like it) but messy and risky if you get lost. Yours sincerely. -- ------------------------------------------------------------------------ Stanley Hopcroft ------------------------------------------------------------------------ '...No man is an island, entire of itself; every man is a piece of the continent, a part of the main. If a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as if a manor of thy friend's or of thine own were. Any man's death diminishes me, because I am involved in mankind; and therefore never send to know for whom the bell tolls; it tolls for thee...' from Meditation 17, J Donne. --------------------------------- Do You Yahoo!? HotJobs, a Yahoo! service - Search Thousands of New Jobs --0-1857066914-1028933138=:31770 Content-Type: text/html; charset=us-ascii Dear Sir, Thx for the solution! But im unable to decipher the command here.. perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log is it necessary that it has to be in multiples of 3600 (1 hr) also pls tell me where this command has to be executed. Also, I was unable to find any relations between the 10 digit time stamps (eg. 10288....), and the corresponding time value. Please help me to know the logic behind the generation of these time stamps. I dont find any common link between the sequence of the time stamps. Is it using some knind of algorithm? Will manual edition help, once the relation is known? PS : I got 2 log files automatically generated for that day (8 august), one with the name.. netsaint-08-08-2002-00.log and other with netsaint-08-08-2002-18.log The system date change was approximately 18-19 hrs ahead of the original time. Is the value 18 in the 2nd log file to do anything with it? your kind help is appreciated. Warm Regds, Amol Sapkal GTL Ltd., Mumbai Stanley Hopcroft wrote: Dear Sir, On Thu, Aug 08, 2002 at 02:20:52PM -0700, amol sapkal wrote: > > Hi All, > > I'm facing a strange but yet significant problem, where in my machine's system timing was changed unknowingly. The NetSaint logs now show me wrong log entries. > > Is there anyway to get back the correct entries? > I thhink that the only way of getting them back is editing them with the swiss army chainsaw, making sure that you are careful not to take your legs off. > Do I need to edit the netsaint logs manually. > perl -i.bak -pe 's#^\[(\d+)\]#$1 - Your_Time_Correction(eg 2*3600)#e' netsaint.log may be something like what you want, provided you know that the log entries are __all__ are out by the same constant amount. (in any case, you should try your auto edit program [if you choose to use one] on a copy of the log. -i.bak will give you the original log with a '.bak' suffix/extension). Otherwise, you'll have to identify when the log got the bad timestamps and then fix that bit. > I have restored the correct system timing now. It really sounds like you must . identify the bad bits in the log . cut them out and save them somewhere . correct them . put the corrected bits back in This is feasable (I have done something like it) but messy and risky if you get lost. Yours sincerely. -- ------------------------------------------------------------------------ Stanley Hopcroft ---------------------------------------------------------------- -------- '...No man is an island, entire of itself; every man is a piece of the continent, a part of the main. If a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as if a manor of thy friend's or of thine own were. Any man's death diminishes >me, because I am involved in mankind; and therefore never send to know for whom the bell tolls; it tolls for thee...' from Meditation 17, J Donne. --------------------------------- Do You Yahoo!? href="http://rd.yahoo.com/careers/mailsig/new/*http://www.hotjobs.com">HotJo bs, a Yahoo! service - Search Thousands of New Jobs --0-1857066914-1028933138=:31770-- -- __--__-- Message: 2 Date: Sat, 10 Aug 2002 09:20:36 +1000 From: Stanley Hopcroft To: [email protected] Subject: [netsaint] Detecting anomalies in time series. Was: checking for glacial changes Dear Ladies and Gentlmen, I am writing with hopefully the last words about this matter. This may be an important matter that my poor words are unfortunately obfuscating. On Fri, Aug 09, 2002 at 07:36:48AM -0400, Bishop, Dean wrote: > maybe you could use something like tcpdump to capture specific network > traffic?? > > This assumes that this is a network app. > Collecting the data is not an issue - sorry, former letters made this unclear. The problem is that for the monitor to make sense of time series data and especially to recognise anomalies. Applications are straightforward :- . InOctet time series: does it look right . Mails/time_units : is the mail server actually working . Syn packets : is this a DOS . Number processed : is the processing system alive ? . VMStat, SAR etc : is the box Ok So given a time series collected by some method and most familiarly represented (for people) as a graph, how does one recognise if the graph looks Ok ? The Cricket people have phrased it better than I can at http://cricket.sourceforge.net/aberrant (There is also a paper by Luca Deri at http://luca.ntop.org/ADS.pdf - Senior Deri is the ntop author - about this matter but it doesn't seem to be specific enough to help me. I probably should read it again) In a nutshell, you have a whole bunch of time series data (that you probably graph), how can one automatically conclude that a dataset indicates a fault ? This I think is a reasonable thing for Netsaint to take an interest in. The Cricket approach to anomaly detection is to . have a modified RRD with a prediction Consolidation function in it . check the actual values against the predicted (and the confidence intervals/error bands) Has anyone had any experience with a time series anomaly detector (Cricket based or otherwise) ? Yours sincerely. -- ------------------------------------------------------------------------ Stanley Hopcroft ------------------------------------------------------------------------ '...No man is an island, entire of itself; every man is a piece of the continent, a part of the main. If a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as if a manor of thy friend's or of thine own were. Any man's death diminishes me, because I am involved in mankind; and therefore never send to know for whom the bell tolls; it tolls for thee...' from Meditation 17, J Donne. -- __--__-- Message: 3 Date: Sat, 10 Aug 2002 13:07:57 +1000 From: Stanley Hopcroft To: [email protected] Subject: Re: [netsaint] Detecting anomalies in time series. Was: checking for glacial changes Dear Ladies and Gentlemen, The development branch of rrdtool (1.1.x) apparently has the new RRAs HWPREDICT (and friends, see the Cricket articles) for anomaly detection in time series. Has anyone any experience with them ? Yours sincerely. -- ------------------------------------------------------------------------ Stanley Hopcroft ------------------------------------------------------------------------ '...No man is an island, entire of itself; every man is a piece of the continent, a part of the main. If a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as if a manor of thy friend's or of thine own were. Any man's death diminishes me, because I am involved in mankind; and therefore never send to know for whom the bell tolls; it tolls for thee...' from Meditation 17, J Donne. -- __--__-- _______________________________________________ Netsaint-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/netsaint-users End of Netsaint-users Digest --__--__-- _______________________________________________ Netsaint-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/netsaint-users End of Netsaint-users Digest --------------------------------- Do You Yahoo!? HotJobs, a Yahoo! service - Search Thousands of New Jobs