Re: FIX classid? ...and TIPS for FIREQOS_MIN_RATE_DIVISOR

Hi Costa,
sorry for the direct send (an oversight).

Thanks for the script, it's perfect for adapt my chart scripts to FireQOS!

I have noted that Netdata is highly optimized, an integration with
NOSQL DB and a web front-end from multiple netdata servers would be
the "missing" part.
Thanks for the explanation. I will study this wonderful project soon.

Have a nice day

---
Sim

2015-01-04 14:33 GMT+01:00 Tsaousis, Costa <[email protected]>:
> Hi Sim,
>
> (please always CC firehol-support so that others may benefit from our
> discussion - thanks)
>
> You can map class ids to class names with this:
>
> ---
>
> #!/bin/bash
>
> dev="$1"
> cmd="cat"
>
> # check FireQOS names for classes
> if [ -f /var/run/fireqos/ifaces/$dev ]
> then
>         name="`cat /var/run/fireqos/ifaces/$dev`"
>         if [ ! -z "$name" -a -f /var/run/fireqos/$name.conf ]
>         then
>                 interface_classes=
>                 . /var/run/fireqos/$name.conf
>                 opts=
>                 for n in $interface_classes_monitor
>                 do
>                         classid=`echo $n | cut -d '|' -f 3`
>                         classname=`echo $n | cut -d '|' -f 2`
>                         opts="$opts -e 's/ $classid / $classname /g'"
>                 done
>                 test ! -z "$opts" && cmd="sed $opts"
>         fi
> fi
>
> tc -s -d class show dev $dev | eval "$cmd"
>
> ---
>
> This will give you the output of tc, but all classids will be replaced
> with class names, as defined in fireqos.conf.
>
> Now, regarding netdata:
>
> Netdata is a highly optimized collector and database for time-series
> values. The charts are just a front-end. They are handled entirely at
> the browser. The server knows nothing about charting.
>
> Check this for example:
>
>    http://www.tsaousis.gr:19999/data/tc.world_out/100/1/average/0/0/nonzero/
>
> It returns 100 rows of data, per second, for all classes of my
> world-out fireqos interface.
>
> This one, returns 50 rows, with average every 60 seconds:
>
>    http://www.tsaousis.gr:19999/data/tc.world_out/50/60/average/0/0/nonzero/
>
> This one is the same, but it gives the max value of every 60 seconds:
>
>    http://www.tsaousis.gr:19999/data/tc.world_out/50/60/max/0/0/nonzero/
>
> Netdata is very efficient. On my celeron J1900 processor it runs, it
> only needs 7ms per second of one core for all data collection it does
> per second. This is 7/1000 of a celeron. For maintaining 7200 seconds
> (2 hours) history in memory for 380 time-series values, it only needs
> 13MB of RAM (I even wrote my own floating point number storage that
> uses 29 bits for the value and 3 bits for various flags related to the
> number).
>
> Even the agregation algorithm is super fast. The 2nd link above took
> 5.71 ms on the server to be calculated (50x60 = 3000 values for 9
> classes = 27.000 numbers averaged).
>
> So, netdata seems already like what you propose. An efficient
> real-time collector on every machine.
>
> I have not added an authontication or access restriction mechanism to
> netdata yet. For the moment if you want to remove the web front end,
> just remove the web files.
>
> Currently I plan to re-factor netdata so that:
>
> 1. Each time-series can be accessed indepedently of the others (what
> you ask for)
> 2. Charts (i.e. collections of time-series values) can be dynamic (so
> that you can say "I want a chart with the voip class of INPUT, the
> voip class of OUTPUT and the sytem time of the CPU)
> 3. It can keep a disk database for longer periods, by storing averaged
> data every 5 mins, 30 mins, 1 hour, etc.
> 4. Integrate events generation to it (i.e. if this number is
> above/bellow this number and that number is above/bellow the other
> number and these conditions are met for 10+ seconds, then fire an
> event to do something).
>
> I do not plan to create a central server. I don't think there will be
> a need for one. My guess is that a web front-end with the capability
> to get data from multiple netdata servers would be better. So each
> server keeps its data, but a single front-end can aggregate and
> combine data from all servers. This will be a lot faster and more
> efficient.
>
> On the other hand, If you really need a history of everything at a
> central place, it would be more efficient to use a NOSQL db to store
> the JSON output of all netdata servers, once every hour (this can be
> easily automated with a script). Then the same front-end could query
> the NOSQL db to get the data it needs for charting.
>
> It is a bit different, but I think a lot better than a huge central
> monitoring system.
>
> Costa
>
>
> On Sat, Jan 3, 2015 at 12:21 PM, Sim <[email protected]> wrote:
>> Hello Costa,
>> thanks for your quick reply!
>>
>> Yes, my idea is to have one monitor server and various remote agent
>> (one for server).
>> In this way I can save/draw multiple charts, for multiple servers, in
>> a unique page/point.
>>
>> For example is possibile to "call" remote agent that it will reply
>> with this output command:
>> /sbin/tc -s -d class show dev $interface (for example using netcat)
>> Here is possible to extract all data: CLASSID - HOST - TC MIN/MAX - TC
>> MIN/MAX - PRIO,
>> value in/out, except the "map for class".
>>
>> Here a real example (with a cron/update evey 1min) and a detail for 6h
>> (rrdtools chart):
>> http://s4.postimg.org/zakm4ldcd/example.png
>> 340 is for me the "note class", the host gp1.tout, min/max from TC and the PRIO:
>>   - = 3
>>  -- = 2
>>   | = 4
>>  + = 5
>> ++ = 6 , etc...
>>
>> Thanks for your tip about Netdata. It's really nice! :-) I have just
>> two questions:
>>
>> Is it possibile to "protect" access? (auth and/or IP filter)
>> In "router" servers I've not iptables or other filter, and the unique way is
>> using hosts.allow/deny or filter connection by service.
>>
>> As my example, my idea (monitor server) is to separate all charts for
>> every CLASS/QoS IN+OUT entry.
>> Is it possibile with Netdata?
>>
>> The great goals would be your Netdata client/server (agent over remote
>> server, with IP source/other protection)
>> and a customizable central server (where select charts to draw in a
>> unique customizable dashboard)
>>
>> Science fiction? Sure... but it would be a good and unique tool for this :-)
>>
>> PS: as in other my emails... always thanks for your Firehol.... I'm
>> using it from 2004!
>>
>> ---
>> Sim
>>
>> 2015-01-02 19:04 GMT+01:00 Tsaousis, Costa <[email protected]>:
>>> Hi Sim, happy new year!
>>>
>>> So you want to give classids yourself? Is this the fix you need?
>>>
>>> Keep in mind that fireqos saves a file per interface in /run/fireqos
>>> that has all the maps required to get class names.
>>> I have used this in netdata (https://github.com/ktsaou/netdata), a
>>> tool I wrote in C for real time monitoring of linux hosts.
>>>
>>> Netdata demo at: http://www.tsaousis.gr:19999/
>>> Scroll down to the QoS section to see FireQoS classes visualized in
>>> realtime. I wrote netdata because all the graphing tools I found are
>>> averaging data every 5 mins. Netdata visualizes everything per second
>>> (zoom on a chart to get per second realtime moniroting), while
>>> requiring very few resources!
>>>
>>> Regards,
>>>
>>> Costa
>>>
>>>
>>> On Thu, Jan 1, 2015 at 10:47 PM, Sim <[email protected]> wrote:
>>>> Happy 2015!
>>>>
>>>> I'm testing FireQOS... great tool... very very thanks Costa!
>>>> Before it I was using "htb" (writing the rules manually or with other tools)
>>>>
>>>> I've a question....
>>>>
>>>> Is possible to "fix" classid?
>>>>
>>>> It is really helpful to identify the class or building charts (with
>>>> RRD for example), without losing the "index" in case of changes
>>>> (add/delete before/after).
>>>>
>>>> PS: This is useful for distributing all the guaranteed minimum
>>>> bandwidth to all classes (if not specified manually):
>>>>
>>>> FIREQOS_MIN_RATE_DIVISOR=`cat /etc/firehol/fireqos.conf | grep class |
>>>> grep -v FIREQOS_MIN_RATE_DIVISOR | sort | uniq | wc -l`
>>>>
>>>> ---
>>>> Sim
>>>> _______________________________________________
>>>> Firehol-support mailing list
>>>> [email protected]
>>>> http://lists.firehol.org/mailman/listinfo/firehol-support