On the road to Xen : LTTng and relay CPU hotplug support

Hi,

In my current work of porting LTTng to the Xen hypervisor, I notice that their
project use the Linux CPU hotplug feature extensively in the domain-0 and
domain-Us. This brings me to a preliminary adaptation phase of the LTTng kernel
tracer : adding CPU hotplug support.

The basic idea is that we have (at least) to create a new buffer for every
channel when a CPU is "UP_PREPARE". In order to do that, Relay must be
modified so the buffer creation function is less coupled with the channel
creation and that this function is exported as part of the API.
(I just implemented this modification) We can also destroy the buffer when the
CPU goes down (see below).

Relay cannot, by itself, deal with the CPU hotplug events because it does not
have a list of the allocated channels. This is why I choose to put the hotplug
event handler inside LTTng. LTTng must also have proper synchronization between
each calls that results in a for_each_online_cpu and trace list modification
(this trace list is used in the CPU hotplug handler to iterate on each channel).

I will also have to add inotify support to lttd so it discovers the creation of
new channel buffer files.

However, there is a design choice that I have to do. Here is the dilemma : we
can either :

A - create new buffers when a new CPU appears and _not_ flush/destroy it when
  the CPU goes down.
  
  This choice imply that we have to iterate on each_possible_cpu and verify if
  the buffer is allocated when we do operations on the trace.
  It will also consume unnecessary memory in cases where a system starts
  tracing with many CPUs and later put some of them offline (very plausible
  in Xen).

B - create new buffers when a new CPU appears and flush/destroy it when a CPU
  goes down.

  This implies that we can iterate on each_online_cpu at any time, but it has
  some other implications :
  If we think of a scenario where the CPU goes up/down/up really fast on a
  machine while a trace is taken, it will result in the possible scenario where
  the buffer is unreferenced by LTTng, still referenced by the consumer (lttd)
  and the CPU coming up wants to recreate it (both the data structures and the
  debugfs file). Clearly, this operation should never fail. A solution around
  this would be to have to ability to "recycle" the buffers when they are
  referenced by the consumer, but not by the writer anymore. Care must be taken
  about things like the "finalized" flag, atomicity, whether lttd will drop the
  reference and must be inotified of the new file, ...
  
  Another problem is when a CPU would have a short life-time : going up/tracing
  a few events/going down. If the consumer (lttd) did not get the chance to
  receive the inotify and therefore cannot open the buffer file, the data will
  never be written to disk before the buffer is destroyed.

  It will also have a performance cost when a cpu is going up/down repeatedly.
  One thing we could do to make this performance hit less problematic is to do
  a lazy buffer removal : X seconds after the CPU is down, we destroy the
  buffer.


What I could do is to implement (A), keeping in mind that the proper way to do
it would be (B), the longer term target therefore being (B). In the mindset of
doing incremental changes that "works", I think (A) would be a good start.

Having your insight on this would be appreciated,

Thanks,

Mathieu


OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68