Re: Nessus scripts and Moore's Law

On Sun, Nov 14, 2004 at 02:07:09AM +0100, Pavel Kankovsky wrote:
> On Fri, 12 Nov 2004, Renaud Deraison wrote:
> 
> > > *If* the VM really speeds things up. Does anybody know how I could
> > 
> > Not "speed things up". Lower the CPU usage.
> 
> Lowering CPU usage is pointless unless you want 1. to run other CPU 
> intensive tasks on the same machine, or 2. to save some joules of 
> electric energy, or 3. want to "speed things up" and CPU is the 
> bottleneck. :)

I want (1): run more processes at the same time. If we can lower the CPU
usage by 50%, assuming that the network connection is not the
bottleneck, we could test twice as many hosts at the same time (and then
there are ways to optimize the bandwidth usage anyway).

> > The problem of the original HTTP caching mecanism is that it would grow
> > the KB, which would in turn make forking way too slow on Linux/FreeBSD
> > systems (where the fork() time is proportional to the process size).
> 
> How much memory did it consume? Yes, you pay for every page whose entry
> must be copied (and prepared for COW if it is writable) but there are
> already hundreds of pages to be duplicated for the program itself, 
> dynamic libraries etc.

I don't have exact figures - it worked fine for me but Michel attempted to
strangle me because it slowed down his laptop to its knees during an
audit, so I disabled it for now and I'm thinking of a better approach.

But this is definitely something I want to investigate.


> On Sat, 13 Nov 2004, Renaud Deraison wrote:
> 
> > Just a side note on this : what takes time is the loading of the .nasl
> > file from disk to memory. The way the interpretor is done today, a
> > compiled .nasl file is actually bigger than a un-compiled one, so the
> > CPU time gained by using a binary file is lost in system calls to read
> > the file from disk.
> 
> Do you need more than 3 syscalls to read the whole file: open(), read(),
> and close()?

No, but the kernel has to actually read the file from disk - and this is
slow.

When I had the plugin-server running (one process loading all the
plugins in memory and "handing them out" to other processes), the system
load was lower (less open()/read()/close() calls) but the results were
not entirely there yet (and these changes bring a lot of complexity to
the behavior of Nessus, so I decided to remove them). However, this
approach, mixed with other optimizations, might prove to be worthwhile
(especially if we use a real VM, where the compilation of the scripts
will probably be slower).


> BTW: One obvious optimization: rewrite recv_line() to read input in bigger
> chunks rather than sucking it by single characters. 

That's done already, using buffered streams (recv_line() calls
read_stream_connection_min() which reads big chunks of data and
bufferizes them).


> Yet another possible optimization: the hashing function in kb.c is
> suspicious (repeated use of << shifts startings characters into oblivion
> when it gets a long key) and might lead to uneven distribution of keys.

I'll have a look at it, but since the number of keys is low anyway, I'm
not too concerned about this (it won't have a significant performance
impact).



				-- Renaud
_______________________________________________
Nessus-devel mailing list
[email protected]
http://mail.nessus.org/mailman/listinfo/nessus-devel