Re: compressible?

* Johnicholas Hines ([email protected]) wrote:
> Hi,
> 
> I was noticing that the (binary) cpu_0 file that lttng generates seems
> to be pretty compressible.
> For example, using gzip, I compressed three cpu_0 files:
> 25M -> 4.2M
> 57M -> 8.3M
> 56M -> 8.2M
> 
> Do you think including a compression step would be useful for
> minimizing disk traffic due to the instrumentation? The potential
> benefit is seems to be a factor of five reduction in disk traffic, but
> I don't know how big the costs are.
> 
> It's also possible my traces are particularly compressible because of
> patterns in what I'm doing.
> 

Hi Johnicholas,

I noticed the same thing : gzip or bzip2 do a pretty good job at compressing the
the traces. It is very useful when it comes to sending them over a slow link.
The nature of traces is to record events repeatedly, which often creates event
records that only differ by their time stamp.

A while ago (5 years ago actually), I worked on a prototype that would add the
ability to compress the traces "live" within the original LTT. I used a modified
hash table (that overwrote the entries upon collision) to keep a table of the
recent events. When an entry had to be replaced, I simply wrote the index and
the new data in the trace, followed by the "real" event : a reference to the
table entry. Each successful hit only had to write the timestamp and table
index. I don't remember the numbers, but it shrunk the traces enough to be
interesting (at least a factor of 2). Additionnaly, this approach does not
consume much CPU time.

The problem with using bzip2 or gzip is the CPU consumption : we have to
determine what will disturb the less the traced kernel : writing a lot of
data to disk or compressing a trace and writing the result to disk. It depends
on what the user is tracing : something CPU hungry or I/O intensive.

In those tests, we have to take in account that lttd does not flush the buffers
to disk : we must therefore create a trace bigger that the available memory on
the system.

The idea is : get the maximum throughput of the hard disk, multiply this
throughput by the compression ratio of bzip2 (this is what the system should be
able to cope with under the hypothesis that the disk I/O is the bottleneck)
and then see how much CPU time bzip2 takes to compress that much data. From our
throughput in MB/s units, we know how much % of cpu time bzip2 would take : if
bzip2 takes more than 100% CPU time, we know that it has become the new
bottleneck. If it takes a small amount of CPU time, then it might be good to
consider using it during tracing.

Compressing (bzip2) a 106MB tracefile (firefox startup) : 42.167s with ~100% cpu
File compressed : 26MB
Compression ratio : 1:24
CPU time : 2.51 MB/s at 100% cpu -> 0.397 s/MB

Hard disk : Model: WDC WD1600JD-00H : specs : buffer-to-disk : 93.5MB/s
(let's take the specs as the upper bound for disk transfer speed)

Using the 1:24 compression ratio, we can take 2244MB/s worth of data at bzip2
input. If it takes 0.397 s of CPU time for each MB of input, it would then take
890.868 s of CPU time _per second_ to compress the quantity of data required to
fill the hard disk I/O.

So, clearly, bzip2 becomes the bottleneck. Using an offline compression, when
necessary, it a good idea, but it hurts performances badly if done online. We
could still think of using a compression algorithm specialized within LTTng, as
the one I explained earlier, which has a small performance impact and still
saves a lot of space.

Mathieu


> Johnicholas
> _______________________________________________
> Ltt-dev mailing list
> [email protected]
> http://listserv.shafik.org/mailman/listinfo/ltt-dev
> 
OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68