Re: CPU Frequency scaling with LTTng

On Mon, 2007-11-12 at 13:14 -0500, Mathieu Desnoyers wrote:
> * Richard Purdie ([email protected]) wrote:
> > On Sun, 2007-11-11 at 19:58 -0500, Mathieu Desnoyers wrote:
> > > > Are there any plans to support frequency scaling and is this approach
> > > > along the right lines? Any thoughts on the subject are welcome!
> > > 
> > > I had a design idea for this, just did not have time to implement it. It
> > > implies modification of both LTTng and LTTV :
> > > 
> > > - We should create a new channel "frequency" channel, which would always
> > > be in "normal" mode (just like the facilities channel). Therefore, we
> > > can read the frequency channel before reading the rest of the trace and
> > > extract the information about each frequency change that happened during
> > > tracing. We can then create a data structure that will map :
> > > 
> > > cycles and cpu number -> time
> > > 
> > > Events in this tracefile will have, just like any event, a timestamp
> > > and, for event data, the new frequency.
> > > 
> > > In LTTng, we will need to record the frequency in an event that uses the
> > > "force" flag (currently used for 64 bits heartbeat written just before
> > > we activate tracing). It separates the timestamp generation from the
> > > physical buffer boundaries. It is useful when we plan to do multiple
> > > trace "start/pause/unpause/.../stop".
> > > 
> > > In LTTV, we would keep one binary tree per CPU in a trace, indexed by
> > > cycles. We would create this tree by reading the frequency channel on a
> > > per cpu tracefile basis, keeping a counter of the time increasing (in
> > > ns), at the rate of the last frequency encountered.
> > > 
> > > Afterward, a query to this data structure would look like :
> > > 
> > > LttTime get_timestamp_from_cpu_cycles(guint64 cycles, guint cpu);
> > > 
> > > This O(log(n)) search would be useful when we look for the right block
> > > containing the information looked for (it's already a O(log(n)) search).
> > > However, for the sequential trace read, we would hook on the frequency
> > > change events in ltt/tracefile.c (it would be an hardcoded event) and
> > > keep track of the current frequency of each CPU. We could then compute
> > > the current time in ns with minimal CPU time required.
> > > 
> > > What do you think of this ?
> > 
> > In trying the approach I mentioned I was keeping in mind the past
> > discussions we've had where you wanted to keep timestamps simple and not
> > need to rely on logged data to be able to interpret the timestamps
> > themselves. Whilst that has been tricky on systems with limited TSCs, I
> > think we've achieved a lot there and it works well and efficiently. 
> > 
> > Ideally, it would be good to be able to take a subbuffer on its own and
> > be able interpret the timestamp data contained within without external
> > references. If the start and end frequencies are in the header as per
> > the existing format its possible to do that.
> > 
> 
> We still have problems synchronizing different buffers together (across
> CPUs on SMP, different channels on UP).

Sorry for the delay in replying to this, I keep getting switched between
projects and you've asked some tricky questions I'm not sure I have
answers to! :).

I'm not sure there is a problem synchronising different channels on UP
since they're all derived from the same tsc. The problem for SMP is
difficult though.

> > Perhaps the way forward would be to allow the "physical" subbuffer to be
> > further split into virtual sections. At each frequency change, we write
> > a length to the active header effectively closing it and start a new
> > header containing the frequency for the new data? Each physical buffer
> > could then contain multiple virtual sections each at a given frequency.
> > 
> > This would allow a virtual "buffer switch" at each frequency change
> > without the overhead of a physical buffer change.
> > 
> > Would that work?
> > 
> 
> The buffers are used to perform fast O(log(n)) time seek in LTTV. It
> relies on the fact that the timestamps are known at the physical buffer
> boundary to do this. I am not sure this concept of "virtual" buffer
> within the physical buffer will give that to us.

It probably would need a field adding to the physical buffer header
signifying how much physical time that buffer represented...

> > I do like the simplicity of the current LTT timestamp approach and am
> > just nervous about anything that complicates it too much!
> > 
> 
> I agree : we must keep this as "dumb-stupidly-reviewable" as possible :)
> What's the point in precise tracing information if we can't trust the
> timestamps.

right :)

> Which brings me to a problem that we will have to face to support
> frequency scaling efficiently on SMP : since there is a delay between
> the CPU frequency change and the moment the kernel is informed
> (very likely interrupt latency + cycles spent in ISR before the event),
> we have to deal with the fact that we will have a false knowledge of the
> CPU frequency for the period that precedes the event. As time goes by,
> this delta between CPUs will get worse.

Yes, any frequency change event means your timestamp conversion is going
to become inaccurate since you don't know how long the frequency change
takes and exactly how many cycles were spent at the start/finish
frequencies.

> Also, we have to think of a way to synchronize the time bases before
> tracing starts. Currently, we are synchronized on the CPU's "0" cycle
> (last time their TSC has been reset). However, since we don't keep track
> of the frequency change history _before_ tracing starts (or should we
> ?),

I was experimenting with some patches to LTTV and actually patched the
start time as 0 as nothing else really makes sense. My personal opinion
is that if people want a meaningful start time, data from any system RTC
should be logged as an event and used to offset the data, kind of in the
same way that statedump is used to correct process data.

Yes, RTC data isn't highly accurate but it is stable over long periods
and won't suffer from the inaccuracies due to frequency changes
mentioned above.

I can't help wondering if there is some way an RTC could be used to at
least detect inaccuracies due to frequency changes. Would it be good
enough to detect the problem and partially correct (with a known maximum
error)?

>  we would have to know as precisely as possible what tsc values are
> equivalent right before we start tracing. Sadly, this operation is
> costly and much more precise when done on an idle system; our results
> could be much worse that the synchronization the kernel does at boot
> time when the CPUs are mostly idle. See ltt/ltt-test-tsc.c to see my
> attempt at doing a less intrusive test of TSC synchronicity. (I tried
> not to disable interrupts for too long)
> 
> We could have to think about redoing this periodically to resynchronize
> our trace timestamps, thus getting an upper bound on the timestamp
> error. However, we would have to make sure this is not too intrusive
> wrt interrupt latency.
> 
> I agree that we would have to write the timestamps and current frequency
> at buffer boundaries, so we can easily seek if we have missing buffers.
> However, we would rely on having a separate channel to keep track of
> frequency changes across the lost buffers to be able to keep them
> synchronized. This is why I suggest to keep this information in a
> separate low traffic channel.

I understand the reason for it, what concerns me is that a separate
channel doesn't really help the synchronisation of the CPUs when
frequency changes are involved without these TSC resync periods. The
synchronisation is one of two main drawback of my simpler approach (the
other is the potentially rapid sub-buffer switching).

It guess it all depends what kind of error we accept in the
synchronisation really. If you consider a 128 way NUMA system with
variable frequency processor nodes separated by slow links, you can
never expect to read the TSC on each processor at the same time :(. They
could all query some reference though to provide a maximum error...

I have a need to address the UP case so I'm going to look at patching
lltv to interpolate based on the buffer headers and patching the kernel
to generate the correct headers. From what I've seen that shouldn't be
too difficult.

Regards,

Richard