Re: CPU Frequency scaling with LTTng

Hi Mathieu,

On Sun, 2007-11-11 at 19:58 -0500, Mathieu Desnoyers wrote:
> * Richard Purdie ([email protected]) wrote:
> > I've been experimenting with CPU frequency scaling and its effect on
> > LTTng. I notice some of the header fields hint at planned support for
> > frequency scaling but the userland tools (LTTV) don't seem to support
> > it?
> > 
> 
> Yes, this is something we have to address. LTTng should live correctly
> with frequency scaling in UP systems, but won't give the correct elapsed
> time in ns.

Yes, incorrect elapsed time and bunching of events which makes reading
the LTT data much tricker.

> In SMP, the current version will detect non synchronized TSCs (which
> happens when frequency scaling is used on x86, x86_64) and will fall
> back on an hybrid clock based on the highest TSC count seen in the
> system and a logical clock.

Right, the SMP case is tricky...

> > >From a kernel point of view the easiest way to handle it seemed to be to
> > force a subbuffer switch immediately after the frequency change. This is
> > relatively straightforward to do with minor alterations to existing
> > code.
> > 
> 
> Hrm, but you can end up having maaaany buffers. How frequent can the
> frequency change be in the worse case ? 

I suspect the answer is very frequent in theory. In practise the
governor smooths them out so I'd expect a maximum frequency of say 1-4Hz
though.

I am using a different transport where the overhead involved in a buffer
switch is less significant but I take your point.

> The idea is interesting and points out a basic need : we have to be able 
> to extract frequency change events efficiently.
[...]
> > 
> > Are there any plans to support frequency scaling and is this approach
> > along the right lines? Any thoughts on the subject are welcome!
> > 
> 
> I had a design idea for this, just did not have time to implement it. It
> implies modification of both LTTng and LTTV :
> 
> - We should create a new channel "frequency" channel, which would always
> be in "normal" mode (just like the facilities channel). Therefore, we
> can read the frequency channel before reading the rest of the trace and
> extract the information about each frequency change that happened during
> tracing. We can then create a data structure that will map :
> 
> cycles and cpu number -> time
> 
> Events in this tracefile will have, just like any event, a timestamp
> and, for event data, the new frequency.
> 
> In LTTng, we will need to record the frequency in an event that uses the
> "force" flag (currently used for 64 bits heartbeat written just before
> we activate tracing). It separates the timestamp generation from the
> physical buffer boundaries. It is useful when we plan to do multiple
> trace "start/pause/unpause/.../stop".
> 
> In LTTV, we would keep one binary tree per CPU in a trace, indexed by
> cycles. We would create this tree by reading the frequency channel on a
> per cpu tracefile basis, keeping a counter of the time increasing (in
> ns), at the rate of the last frequency encountered.
> 
> Afterward, a query to this data structure would look like :
> 
> LttTime get_timestamp_from_cpu_cycles(guint64 cycles, guint cpu);
> 
> This O(log(n)) search would be useful when we look for the right block
> containing the information looked for (it's already a O(log(n)) search).
> However, for the sequential trace read, we would hook on the frequency
> change events in ltt/tracefile.c (it would be an hardcoded event) and
> keep track of the current frequency of each CPU. We could then compute
> the current time in ns with minimal CPU time required.
> 
> What do you think of this ?

In trying the approach I mentioned I was keeping in mind the past
discussions we've had where you wanted to keep timestamps simple and not
need to rely on logged data to be able to interpret the timestamps
themselves. Whilst that has been tricky on systems with limited TSCs, I
think we've achieved a lot there and it works well and efficiently. 

Ideally, it would be good to be able to take a subbuffer on its own and
be able interpret the timestamp data contained within without external
references. If the start and end frequencies are in the header as per
the existing format its possible to do that.

Perhaps the way forward would be to allow the "physical" subbuffer to be
further split into virtual sections. At each frequency change, we write
a length to the active header effectively closing it and start a new
header containing the frequency for the new data? Each physical buffer
could then contain multiple virtual sections each at a given frequency.

This would allow a virtual "buffer switch" at each frequency change
without the overhead of a physical buffer change.

Would that work?

I do like the simplicity of the current LTT timestamp approach and am
just nervous about anything that complicates it too much!

Regards,

Richard