Re: CPU Frequency scaling with LTTng

* Richard Purdie ([email protected]) wrote:
> Hi Mathieu,
> 
> On Sun, 2007-11-11 at 19:58 -0500, Mathieu Desnoyers wrote:
> > * Richard Purdie ([email protected]) wrote:
> > > I've been experimenting with CPU frequency scaling and its effect on
> > > LTTng. I notice some of the header fields hint at planned support for
> > > frequency scaling but the userland tools (LTTV) don't seem to support
> > > it?
> > > 
> > 
> > Yes, this is something we have to address. LTTng should live correctly
> > with frequency scaling in UP systems, but won't give the correct elapsed
> > time in ns.
> 
> Yes, incorrect elapsed time and bunching of events which makes reading
> the LTT data much tricker.
> 
> > In SMP, the current version will detect non synchronized TSCs (which
> > happens when frequency scaling is used on x86, x86_64) and will fall
> > back on an hybrid clock based on the highest TSC count seen in the
> > system and a logical clock.
> 
> Right, the SMP case is tricky...
> 
> > > >From a kernel point of view the easiest way to handle it seemed to be to
> > > force a subbuffer switch immediately after the frequency change. This is
> > > relatively straightforward to do with minor alterations to existing
> > > code.
> > > 
> > 
> > Hrm, but you can end up having maaaany buffers. How frequent can the
> > frequency change be in the worse case ? 
> 
> I suspect the answer is very frequent in theory. In practise the
> governor smooths them out so I'd expect a maximum frequency of say 1-4Hz
> though.
> 
> I am using a different transport where the overhead involved in a buffer
> switch is less significant but I take your point.
> 
> > The idea is interesting and points out a basic need : we have to be able 
> > to extract frequency change events efficiently.
> [...]
> > > 
> > > Are there any plans to support frequency scaling and is this approach
> > > along the right lines? Any thoughts on the subject are welcome!
> > > 
> > 
> > I had a design idea for this, just did not have time to implement it. It
> > implies modification of both LTTng and LTTV :
> > 
> > - We should create a new channel "frequency" channel, which would always
> > be in "normal" mode (just like the facilities channel). Therefore, we
> > can read the frequency channel before reading the rest of the trace and
> > extract the information about each frequency change that happened during
> > tracing. We can then create a data structure that will map :
> > 
> > cycles and cpu number -> time
> > 
> > Events in this tracefile will have, just like any event, a timestamp
> > and, for event data, the new frequency.
> > 
> > In LTTng, we will need to record the frequency in an event that uses the
> > "force" flag (currently used for 64 bits heartbeat written just before
> > we activate tracing). It separates the timestamp generation from the
> > physical buffer boundaries. It is useful when we plan to do multiple
> > trace "start/pause/unpause/.../stop".
> > 
> > In LTTV, we would keep one binary tree per CPU in a trace, indexed by
> > cycles. We would create this tree by reading the frequency channel on a
> > per cpu tracefile basis, keeping a counter of the time increasing (in
> > ns), at the rate of the last frequency encountered.
> > 
> > Afterward, a query to this data structure would look like :
> > 
> > LttTime get_timestamp_from_cpu_cycles(guint64 cycles, guint cpu);
> > 
> > This O(log(n)) search would be useful when we look for the right block
> > containing the information looked for (it's already a O(log(n)) search).
> > However, for the sequential trace read, we would hook on the frequency
> > change events in ltt/tracefile.c (it would be an hardcoded event) and
> > keep track of the current frequency of each CPU. We could then compute
> > the current time in ns with minimal CPU time required.
> > 
> > What do you think of this ?
> 
> In trying the approach I mentioned I was keeping in mind the past
> discussions we've had where you wanted to keep timestamps simple and not
> need to rely on logged data to be able to interpret the timestamps
> themselves. Whilst that has been tricky on systems with limited TSCs, I
> think we've achieved a lot there and it works well and efficiently. 
> 
> Ideally, it would be good to be able to take a subbuffer on its own and
> be able interpret the timestamp data contained within without external
> references. If the start and end frequencies are in the header as per
> the existing format its possible to do that.
> 

We still have problems synchronizing different buffers together (across
CPUs on SMP, different channels on UP).

> Perhaps the way forward would be to allow the "physical" subbuffer to be
> further split into virtual sections. At each frequency change, we write
> a length to the active header effectively closing it and start a new
> header containing the frequency for the new data? Each physical buffer
> could then contain multiple virtual sections each at a given frequency.
> 
> This would allow a virtual "buffer switch" at each frequency change
> without the overhead of a physical buffer change.
> 
> Would that work?
> 

The buffers are used to perform fast O(log(n)) time seek in LTTV. It
relies on the fact that the timestamps are known at the physical buffer
boundary to do this. I am not sure this concept of "virtual" buffer
within the physical buffer will give that to us.


> I do like the simplicity of the current LTT timestamp approach and am
> just nervous about anything that complicates it too much!
> 

I agree : we must keep this as "dumb-stupidly-reviewable" as possible :)
What's the point in precise tracing information if we can't trust the
timestamps.

Which brings me to a problem that we will have to face to support
frequency scaling efficiently on SMP : since there is a delay between
the CPU frequency change and the moment the kernel is informed
(very likely interrupt latency + cycles spent in ISR before the event),
we have to deal with the fact that we will have a false knowledge of the
CPU frequency for the period that precedes the event. As time goes by,
this delta between CPUs will get worse.

Also, we have to think of a way to synchronize the time bases before
tracing starts. Currently, we are synchronized on the CPU's "0" cycle
(last time their TSC has been reset). However, since we don't keep track
of the frequency change history _before_ tracing starts (or should we
?), we would have to know as precisely as possible what tsc values are
equivalent right before we start tracing. Sadly, this operation is
costly and much more precise when done on an idle system; our results
could be much worse that the synchronization the kernel does at boot
time when the CPUs are mostly idle. See ltt/ltt-test-tsc.c to see my
attempt at doing a less intrusive test of TSC synchronicity. (I tried
not to disable interrupts for too long)

We could have to think about redoing this periodically to resynchronize
our trace timestamps, thus getting an upper bound on the timestamp
error. However, we would have to make sure this is not too intrusive
wrt interrupt latency.

I agree that we would have to write the timestamps and current frequency
at buffer boundaries, so we can easily seek if we have missing buffers.
However, we would rely on having a separate channel to keep track of
frequency changes across the lost buffers to be able to keep them
synchronized. This is why I suggest to keep this information in a
separate low traffic channel.

Mathieu


> Regards,
> 
> Richard
> 
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68