Rabins and netflow 5 failure

Richard Rothwell via Argus-info <[email protected]>
Newsgroups gmane.network.argus
Message-ID <[email protected]>
Hi Carter,

I attempted to use a netflow 5 stream instead of netflow 9 as input into rabins.
This is causing problems.

As background:
I rewrote InSight’s rabins output data parser to batch the rabins output by line count, rather than by management STOP records.
This was intended to work around the problem with infrequent Argus STOP records when rabins processes Netflow 5.
This seems to have successful.
The new parser works with both Netflow 5 and Netflow9 streams.

When using Neflow 5 there are 2 problems:

1.       The timestamps of the rabins Argus records seem to be wrong. They are piled up in a narrow range of time.

2.       Rabins dies after about 20 minutes of processing.

When using Netflow 9 everything is OK.
It runs indefinitely and the timestamps are fine.

I recompiled rabins with debugging on and executed it as follows:
/usr/local/bin/rabins -D 10 -S 10.169.13.231:562 -M time 60s -B 60s -f /db/farm/elephant/racluster.conf -F /db/farm/elephant/rarc > rabins_netflow5.out 2> rabins_errors.txt &

This produces a 1.5Gig file. The last time anything interesting happens is at the 98% point in the file. There’s just lots of ArgusFree after that.
I am attaching an extract at that point.
There is no obvious issue apart from a queue running out of content.

Regards
rabins_stderr.txt (text/plain, 16.4 KB)
rabins[29382.00b7d6590d7f0000]: 1467936409.523476 ArgusFree (0x6ae5530)
rabins[29382.00b7d6590d7f0000]: 1467936409.523484 ArgusFree (0x6ae5190)
rabins[29382.00b7d6590d7f0000]: 1467936409.523492 ArgusDeleteRecordStruct (0x7f0d59c26010, 0x6ae5190)
rabins[29382.00b7d6590d7f0000]: 1467936409.523499 ArgusPopQueue (0x2493d90) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.523508 ArgusMalloc (8) returning 0x6ae5530
rabins[29382.00b7d6590d7f0000]: 1467936409.523516 ArgusSortQueue(0x24943f0) returned
rabins[29382.00b7d6590d7f0000]: 1467936409.523524 ArgusPopQueue (0x24943f0) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.523533 ArgusMalloc (8) returning 0x6ae5510
rabins[29382.00b7d6590d7f0000]: 1467936409.523540 ArgusSortQueue(0x24947e0) returned
rabins[29382.00b7d6590d7f0000]: 1467936409.523548 ArgusPopQueue (0x24947e0) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.523571 ArgusCalloc (1, 384) returning 0x6de7290
rabins[29382.00b7d6590d7f0000]: 1467936409.523581 ArgusCalloc (1, 128) returning 0x342d400
rabins[29382.00b7d6590d7f0000]: 1467936409.523590 ArgusGenerateStatusMarRecord(0x0, 48) returning 0x6de7290
rabins[29382.00b7d6590d7f0000]: 1467936409.523599 ArgusAdjustGlobalTime real 1467936409.523599 global 1467936409.523599
rabins[29382.00b7d6590d7f0000]: 1467936409.523608 ArgusPrintProto (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523618 ArgusPrintState (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523627 ArgusPrintSourceID (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523639 ArgusPrintStartDate (0x7fff6c7174d0, 0x6de7290, 17)
rabins[29382.00b7d6590d7f0000]: 1467936409.523651 ArgusPrintLastDate (0x7fff6c7174d0, 0x6de7290 17)
rabins[29382.00b7d6590d7f0000]: 1467936409.523661 ArgusPrintDuration (0x7f0d59c26010, 0x7fff6c7174d0, 0x6de7290, 8)
rabins[29382.00b7d6590d7f0000]: 1467936409.523671 ArgusPrintSrcAddr (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523680 ArgusPrintDirection (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523689 ArgusPrintDstAddr (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523698 ArgusPrintSrcPort (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523707 ArgusPrintDstPort (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523716 ArgusPrintPackets (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523725 ArgusPrintBytes (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523738 ArgusPrintAppBytes (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523748 ArgusPrintPercentLoss (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523757 ArgusPrintSrcMacAddress (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523766 ArgusPrintDstMacAddress (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523775 ArgusPrintSrcVID (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523783 ArgusPrintDstVID (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523792 ArgusPrintSrcJitter (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523801 ArgusPrintDstJitter (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523810 ArgusGetIndicatorString (0x7f0d59c26010, 0x6de7290, 0x7fff6c717410)
rabins[29382.00b7d6590d7f0000]: 1467936409.523819 ArgusPrintFlags (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523827 ArgusPrintTransactions (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523836 ArgusPrintTCPOptions (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523846 ArgusPrintSrcAppBytes (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523868 ArgusPrintSrcBytes (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523877 ArgusPrintSrcPackets (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523886 ArgusPrintPercentSrcLoss (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523895 ArgusPrintSrcWindow (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523904 ArgusPrintSrcLoad (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523913 ArgusPrintDstAppBytes (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523921 ArgusPrintDstBytes (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523930 ArgusPrintDstPackets (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523939 ArgusPrintPercentDstLoss (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523948 ArgusPrintDstWindow (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523957 ArgusPrintDstLoad (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523965 ArgusPrintInode (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523975 ArgusPrintTCPRTT (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523984 ArgusPrintSrcCountryCode (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.523992 ArgusPrintDstCountryCode (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.524001 ArgusPrintSrcAsn (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.524010 ArgusPrintDstAsn (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.524018 ArgusPrintProducerConsumerRatio (0x7fff6c7174d0, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.524028 ArgusPrintRecord (0x7f0d59c26010, 0x7fff6c727970, 0x6de7290, 4096)
rabins[29382.00b7d6590d7f0000]: 1467936409.524039 ArgusFree (0x342d400)
rabins[29382.00b7d6590d7f0000]: 1467936409.524047 ArgusFree (0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.524055 ArgusDeleteRecordStruct (0x7f0d59c26010, 0x6de7290)
rabins[29382.00b7d6590d7f0000]: 1467936409.524068 ArgusFree (0x2494760)
rabins[29382.00b7d6590d7f0000]: 1467936409.524076 ArgusPopQueue (0x24947e0) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.524084 ArgusFree (0x6ae5510)
rabins[29382.00b7d6590d7f0000]: 1467936409.524092 ArgusFree (0x24947e0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524100 ArgusDeleteQueue (0x24947e0) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.524254 ArgusEmptyHashTable (0x2494840) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.524298 ArgusFree (0x7f0d58ddf010)
rabins[29382.00b7d6590d7f0000]: 1467936409.524317 ArgusFree (0x2494840)
rabins[29382.00b7d6590d7f0000]: 1467936409.524325 ArgusDeleteHashTable (0x2494840)
rabins[29382.00b7d6590d7f0000]: 1467936409.524334 ArgusFree (0x24944c0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524341 ArgusDeleteAggregator(0x7f0d59c26010, 0x24944c0) returned
rabins[29382.00b7d6590d7f0000]: 1467936409.524350 ArgusFree (0x2494230)
rabins[29382.00b7d6590d7f0000]: 1467936409.524358 ArgusFree (0x24942b0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524365 ArgusFree (0x24942f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524373 ArgusFree (0x2494330)
rabins[29382.00b7d6590d7f0000]: 1467936409.524381 ArgusFree (0x2494370)
rabins[29382.00b7d6590d7f0000]: 1467936409.524388 ArgusFree (0x24943b0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524396 ArgusPopQueue (0x24943f0) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.524415 ArgusFree (0x6ae5530)
rabins[29382.00b7d6590d7f0000]: 1467936409.524424 ArgusFree (0x24943f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524431 ArgusDeleteQueue (0x24943f0) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.524578 ArgusEmptyHashTable (0x2494450) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.524594 ArgusFree (0x7f0d58e60010)
rabins[29382.00b7d6590d7f0000]: 1467936409.524605 ArgusFree (0x2494450)
rabins[29382.00b7d6590d7f0000]: 1467936409.524614 ArgusDeleteHashTable (0x2494450)
rabins[29382.00b7d6590d7f0000]: 1467936409.524622 ArgusFree (0x2493f90)
rabins[29382.00b7d6590d7f0000]: 1467936409.524631 ArgusDeleteAggregator(0x7f0d59c26010, 0x2493f90) returned
rabins[29382.00b7d6590d7f0000]: 1467936409.524639 ArgusFree (0x2494880)
rabins[29382.00b7d6590d7f0000]: 1467936409.524648 ArgusFree (0x2493b90)
rabins[29382.00b7d6590d7f0000]: 1467936409.524656 ArgusFree (0x2493c10)
rabins[29382.00b7d6590d7f0000]: 1467936409.524664 ArgusFree (0x2493c50)
rabins[29382.00b7d6590d7f0000]: 1467936409.524672 ArgusFree (0x2493c90)
rabins[29382.00b7d6590d7f0000]: 1467936409.524680 ArgusFree (0x2493cd0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524689 ArgusFree (0x2493d10)
rabins[29382.00b7d6590d7f0000]: 1467936409.524697 ArgusFree (0x2493d50)
rabins[29382.00b7d6590d7f0000]: 1467936409.524705 ArgusPopQueue (0x2493d90) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.524714 ArgusFree (0x184d8b0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524722 ArgusFree (0x2493d90)
rabins[29382.00b7d6590d7f0000]: 1467936409.524730 ArgusDeleteQueue (0x2493d90) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.524826 ArgusEmptyHashTable (0x2493df0) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.524940 ArgusFree (0x7f0d58ee1010)
rabins[29382.00b7d6590d7f0000]: 1467936409.524952 ArgusFree (0x2493df0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524960 ArgusDeleteHashTable (0x2493df0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524968 ArgusFree (0x24938f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524976 ArgusDeleteAggregator(0x7f0d59c26010, 0x24938f0) returned
rabins[29382.00b7d6590d7f0000]: 1467936409.524985 ArgusFree (0x3777fd0)
rabins[29382.00b7d6590d7f0000]: 1467936409.524993 ArgusFree (0x2493280)
rabins[29382.00b7d6590d7f0000]: 1467936409.525001 ArgusFree (0x2493300)
rabins[29382.00b7d6590d7f0000]: 1467936409.525009 ArgusFree (0x2493340)
rabins[29382.00b7d6590d7f0000]: 1467936409.525016 ArgusFree (0x2493380)
rabins[29382.00b7d6590d7f0000]: 1467936409.525024 ArgusFree (0x24933c0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525032 ArgusFree (0x2493400)
rabins[29382.00b7d6590d7f0000]: 1467936409.525040 ArgusPopQueue (0x2493440) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.525048 ArgusFree (0x6c240b0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525056 ArgusFree (0x2493440)
rabins[29382.00b7d6590d7f0000]: 1467936409.525063 ArgusDeleteQueue (0x2493440) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.525192 ArgusEmptyHashTable (0x24934a0) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.525301 ArgusFree (0x7f0d58f62010)
rabins[29382.00b7d6590d7f0000]: 1467936409.525315 ArgusFree (0x24934a0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525323 ArgusDeleteHashTable (0x24934a0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525336 ArgusFree (0x2492fe0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525344 ArgusDeleteAggregator(0x7f0d59c26010, 0x2492fe0) returned
rabins[29382.00b7d6590d7f0000]: 1467936409.525353 ArgusFree (0x2492cf0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525361 ArgusFree (0x2492d70)
rabins[29382.00b7d6590d7f0000]: 1467936409.525369 ArgusFree (0x2492db0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525376 ArgusFree (0x2492df0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525384 ArgusPopQueue (0x2492e30) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.525392 ArgusFree (0x1c79da0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525400 ArgusFree (0x2492e30)
rabins[29382.00b7d6590d7f0000]: 1467936409.525408 ArgusDeleteQueue (0x2492e30) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.525546 ArgusEmptyHashTable (0x2492e90) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.525561 ArgusFree (0x7f0d58fe3010)
rabins[29382.00b7d6590d7f0000]: 1467936409.525571 ArgusFree (0x2492e90)
rabins[29382.00b7d6590d7f0000]: 1467936409.525579 ArgusDeleteHashTable (0x2492e90)
rabins[29382.00b7d6590d7f0000]: 1467936409.525588 ArgusFree (0x2492a50)
rabins[29382.00b7d6590d7f0000]: 1467936409.525595 ArgusDeleteAggregator(0x7f0d59c26010, 0x2492a50) returned
rabins[29382.00b7d6590d7f0000]: 1467936409.525605 ArgusFree (0x364a700)
rabins[29382.00b7d6590d7f0000]: 1467936409.525612 ArgusFree (0x2492810)
rabins[29382.00b7d6590d7f0000]: 1467936409.525620 ArgusPopQueue (0x2492890) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.525629 ArgusFree (0x15f27d0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525637 ArgusFree (0x2492890)
rabins[29382.00b7d6590d7f0000]: 1467936409.525645 ArgusDeleteQueue (0x2492890) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.525771 ArgusEmptyHashTable (0x24928f0) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.525785 ArgusFree (0x7f0d59064010)
rabins[29382.00b7d6590d7f0000]: 1467936409.525796 ArgusFree (0x24928f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525803 ArgusDeleteHashTable (0x24928f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525811 ArgusFree (0x2492570)
rabins[29382.00b7d6590d7f0000]: 1467936409.525819 ArgusDeleteAggregator(0x7f0d59c26010, 0x2492570) returned
rabins[29382.00b7d6590d7f0000]: 1467936409.525827 ArgusFree (0x24924f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525835 RaDeleteBin(0x24924f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525843 ArgusFree (0x1650a10)
rabins[29382.00b7d6590d7f0000]: 1467936409.525851 ArgusFree (0x16175a0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525860 ArgusPopQueue (0x165b570) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.525868 ArgusFree (0x165b570)
rabins[29382.00b7d6590d7f0000]: 1467936409.525876 ArgusDeleteQueue (0x165b570) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.525885 ArgusFree (0x165e6f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525893 RaParseComplete(caught signal 0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525901 RaParseComplete(0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525909 ArgusShutDown (0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525917 ArgusPopQueue (0x15f2190) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.525925 ArgusFree (0x15f2190)
rabins[29382.00b7d6590d7f0000]: 1467936409.525933 ArgusDeleteQueue (0x15f2190) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.525941 ArgusPopQueue (0x15f21f0) returning 0x7f0d59bb5010
rabins[29382.00b7d6590d7f0000]: 1467936409.525963 ArgusFree (0x7f0d59bb5010)
rabins[29382.00b7d6590d7f0000]: 1467936409.525974 ArgusPopQueue (0x15f21f0) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.525982 ArgusPopQueue (0x15f21f0) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.525990 ArgusFree (0x15f21f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.525998 ArgusDeleteQueue (0x15f21f0) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.526006 RaParseComplete(0)
rabins[29382.00b7d6590d7f0000]: 1467936409.526014 ArgusWindowClose () returning
rabins[29382.00b7d6590d7f0000]: 1467936409.526022 RaParseComplete(caught signal 0)
rabins[29382.00b7d6590d7f0000]: 1467936409.526033 ArgusPopQueue ((nil)) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.526042 ArgusPopQueue ((nil)) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.526365 ArgusFree (0x15f27f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.526380 ArgusFree (0x15f2890)
rabins[29382.00b7d6590d7f0000]: 1467936409.526388 ArgusDeleteModeList () returning
rabins[29382.00b7d6590d7f0000]: 1467936409.526400 ArgusFree (0x15f2050)
rabins[29382.00b7d6590d7f0000]: 1467936409.526408 ArgusDeleteList (0x15f2050, 4) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.526422 ArgusFree (0x15f20f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.526430 ArgusDeleteList (0x15f20f0, 4) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.526438 ArgusFree (0x37ad160)
rabins[29382.00b7d6590d7f0000]: 1467936409.526446 ArgusPopQueue (0x37ad1e0) returning (nil)
rabins[29382.00b7d6590d7f0000]: 1467936409.526454 ArgusFree (0x37ad1e0)
rabins[29382.00b7d6590d7f0000]: 1467936409.526462 ArgusDeleteQueue (0x37ad1e0) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.526590 ArgusEmptyHashTable (0x37ad0e8) returning
rabins[29382.00b7d6590d7f0000]: 1467936409.526604 ArgusFree (0x7f0d58d5d010)
rabins[29382.00b7d6590d7f0000]: 1467936409.526618 ArgusFree (0x4115a60)
rabins[29382.00b7d6590d7f0000]: 1467936409.526626 ArgusFree (0x41158c0)
rabins[29382.00b7d6590d7f0000]: 1467936409.526634 ArgusFree (0x4115b30)
rabins[29382.00b7d6590d7f0000]: 1467936409.526642 ArgusFree (0x4115720)
rabins[29382.00b7d6590d7f0000]: 1467936409.526650 ArgusFree (0x4115990)
rabins[29382.00b7d6590d7f0000]: 1467936409.526658 ArgusFree (0x4115580)
rabins[29382.00b7d6590d7f0000]: 1467936409.526666 ArgusFree (0x41157f0)
rabins[29382.00b7d6590d7f0000]: 1467936409.526673 ArgusFree (0x41153e0)
rabins[29382.00b7d6590d7f0000]: 1467936409.526681 ArgusFree (0x4115240)
rabins[29382.00b7d6590d7f0000]: 1467936409.526689 ArgusFree (0x41154b0)
rabins[29382.00b7d6590d7f0000]: 1467936409.526697 ArgusFree (0x4115650)
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.