Re: [diamon-discuss] Diamon CTF Roadmap?

You can see my answer below preceded by [Dominique Toupin]

-----Original Message-----
From: Wang Nan [mailto:[email protected]] 
Sent: November-25-14 9:08 PM
To: Dominique Toupin; [email protected]; Jiri Olsa
Cc: genghui 00204690; Xinwei Hu
Subject: Re: Diamon CTF Roadmap?

On 2014/11/25 21:42, Dominique Toupin wrote:
> Hi Nan,
> 
> Many tools are able to handle CTF today, e.g.:
> - GDB Tracepoint 
> https://sourceware.org/gdb/onlinedocs/gdb/Trace-Files.html#Trace-Files
> - LTTng kernel and LTTng user space C/C++, Java, Python, etc. 
> https://lttng.org/download
> - babel trace, http://lttng.org/viewers/
> - Trace Compass 
> http://projects.eclipse.org/projects/tools.tracecompass
> - CTF tracer for bare metal 
> https://github.com/efficios/barectf/blob/master/README.md
> 
> You will also find many converter to CTF, e.g. NEXUS HW Trace, perf, 
> And I am most likely missing a few.
> 
> If you can provide a description of your needs it would help to give you more info related to your use case.
> e.g. is it more related to cloud, embedded data base, real time, or do you have a specific view or analysis you want to see in the trace viewer, etc.
> 

Our use cases are diverse, both embedded and server scene have requirements on profiling, and methodologies are also different.

We have two starting points now:

1. On ARM (32/64) platform, extracing information (instruction footprints) from CoreSight
   using perf and use CTF as trace format, to make it possible to analysis with trace from
   upper layer. Intel has similar hardware too, but currently only on specification.

[Dominique Toupin] I believe they are able to get perf data in CTF and if you also have 
CTF trace from upper layer then you can use Trace Compass (python should also be an option) 
to visualize, correlate, analyze trace data from different layers, you can also open text logs in 
Trace Compass. In the upper layer, you can take a similar approach as the Linux kernel where 
static tracepoints represent the wisdom of developers who are most familiar with the code and 
the rest of the world (testers, system administrators, field engineers, developers) can use them 
to extract a great deal of useful information without having to know the code. 

2. On server field, providing some interactive, graphical user interfaces for some generic
   problems. Android's SysTrace is a good example. However, our the real goal is to create
   a tool for dealing with problems not so generic. Profiling using a language like SQL maybe,
   like Splunk, to synthesize traces from different layers and components. So the underlying
   traces are better to be in a uniform format.

[Dominique Toupin] SQL can be nice but it requires a lot more space, if I remember 
correctly, the size is sometimes multiplied by 70 compared to CTF. CTF is like a specialized 
trace data base, in Trace Compass they have efficient query to navigate in the trace, they
 add a state system, etc. You can do all this with SQL but it require a lot more space and it is slower.
If you see things missing in CTF compared to what you want to do with SQL, let us know we can 
investigate how to add those features directly in CTF.


We also concern the upstreaming plan about kernel stuffs in DiaMon.

> Thanks for the second post!
> 
> Dominique
> 
> -----Original Message-----
> From: Wang Nan [mailto:[email protected]]
> Sent: November-25-14 8:04 AM
> To: [email protected]; Dominique Toupin; Jiri 
> Olsa
> Cc: genghui 00204690; Xinwei Hu
> Subject: Diamon CTF Roadmap?
> 
> Hi everyone,
> 
> Looks like my post will be the second thread in this list. That's great.
> 
> I'll be the technical contact person representing Huawei for Diamon.
> I'm glad to work with this new workgroup to create "de-facto, widely used open source trace technologies".
> 
> Beyond sharing new ideas and technologies, our main concern on Diamon now are CTF tracing format and related things for analysing traces from different sources in a unified manner. I have read Jiri Olsa's slides in TracingSummit 2014, and also begin to read his code in https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git. However, it looks like that there's only a trace generating tool is under developing. What about upper trace parsing and analysing tools? Is there any roadmap on it? How can we participate in the project?
> 
> Thanks.
>