Re: Re: conversation history

Something important to remember:  This format is intended to be a 
master format that stores all the information in an easy to parse form. 
  Displaying in a web browser or more fancy display is really the job of 
an XSLT/CSS combination.  I presented this one with a simple CSS for 
the cases where you just want to open the thing really quickly to check 
something without launching Fire.

One more thing I forgot to mention:  This will NOT BE IN 1.0!!!  I will 
put this in the CVS repository shortly after 1.0 is released.  We will 
have that time frame to figure out what exactly we want to do with 
this.

See responses below:


------------------------------------------------------------------
Graham Booker                   Texas A&M University
[email protected]    Graduate Student in ELEN
------------------------------------------------------------------
On Apr 3, 2004, at 10:44 AM, Elektron wrote:

>
> On 3 Apr, 2004, at 23:19, Graham Booker wrote:
>
> And I, being all old-skool and stuff (not really) prefer pure text. Of 
> course, HTML is better since then you can é, so you don't have 
> to worry about color codes and character sets (I'm speaking from an 
> IRC perspective).
>

Note, it doesn't say it yet, but the character set for the logged 
version will be UTF-8.  We do all the character set encoding/decoding 
internally within Fire, and it comes in the form of NSAttributedStrings 
before translated to XML.

> While I like XML, it's bloaty and also a pain to grep (unless you 
> write a log parser, which I'm rarely in the mood to do). Proteus' old 
> logs were XML (a property list, IIRC), which were a pain to read (in 
> less), and a pain to search (since the sender and stuff is on a 
> separate line).
>

It is because of this that I have thought about making a parser that 
could output simple html or text.  It would simply be a matter of 
searching for the <sender> tag, get its content, and tack it on the 
front of each subsequent <message> tag's contents.  That would give you 
a simple HTML.  Removing the tags from the contents would give you 
text.  I don't know if I will ever get to it, but the idea is out 
there.

> For an analogy, I like Java, but it's bloaty and slow (but it's not as 
> bloaty as OSX's packages!).
>
> Proteus then switched to some DB format, which was even worse, since 
> the only thing that could parse it was Proteus, which sucked up around 
> 100 MB to parse 12 MB of logs (usually I closed the window at around 
> 70/80, which stayed in use until I quit proteus). After switching to 
> Fire, I downloaded the source to whatever DB engine they used and 
> exported the SQL. I've yet to code something to split it into multiple 
> files (I started around last year?).

Maybe this will help you:
http://cvs.sourceforge.net/viewcvs.py/adium/adium/Utilities/proteus/

I have to admit, I bounced around the idea of SQL DB, but the issue is 
that you can't access it.  Now if you were running a web server with an 
SQL-capable PHP, this is a non-issue, but how many people out there 
are?  I decided that a file would still be the best format.

>
> That said, the purpose of logs can be split into two things (I think 
> system designers call these specifications):
> 1. To record what happens.
> 2. To refer to later.
>

For this reason, I decided that XML would be a nice change.  I have had 
instances where I needed to separate the sender, the date, and the 
message from the logged format, and as I am sure you know, this is not 
easy to do.

> To do this properly, there are a few requirements (I think they call 
> these design parameters):
> 1a. The format must be unambiguous.
> 1b. Everything must be recorded (if the user so desires).
> 1c. The format must be extendable.
> 2a. The format must be easy to read (e.g. with less).
> 2b. The format must be easy to parse (with whatever log parser).
> 2c. The format must be easy to search (e.g. with grep).
>
> We also probably want it to be opensourcey. This limits us to XML, 
> HTML, or text. Since we want it to be both unambiguous and extendable, 
> we're stuck with XML, unless anyone wants to make up the world's 
> weirdest text format, or HTML with comments/weird attributes/whatever.
>
> Read: XML may be icky, but it's unfortunately the best there is.
>
> I'll make a few suggestions:
> A. Add newlines in appropriate places (after each envelope/event) to 
> make grep easier.

This is easy to do.  In the case of grep, this should be all that is 
really needed.  I will do this then.

> B. Save the ID of the user, e.g. <sender 
> id="[email protected]">meetoo</sender> (or screen name/number/email 
> address as appropriate). For a direct connection (e.g. DCC or 
> whatever), also ip="1.2.3.4".

Yes, more attributes need to be saved.  The plus side of this:  I have 
changed MessageItem (our entire message processing engine in 1.0) to 
contain a dictionary of other attributes.  If a service sets such 
attributes in the message before it is processed, then these attributes 
make it into the XML.  IP should likely go in the event marking the 
direct connection because it doesn't need to be logged each time.  
Regardless, the service will say what attributes it wants in the 
message, and the rest of the code is unchanged to get these attributes 
in there.

> C. Save the ID of the service, e.g. <log began="2004-04-02 22:01:57 
> -0600" protocol="irc" server="irc.nowhere.net"> (or 
> icq/icq.mirabilis.com, etc).

Along these lines, I should likely add in attributes for the message 
item called something like "logAttributes" and "senderAttributes"  Then 
these attributes will make it into the log tag and the sender tag 
respectively.

>
> The following also apply to chat windows:
> D. Save the (server-side) nickname of the user (to track nick 
> changes/MSN nicknames/etc), since it's often more than just a nickname 
> (some people put poems there).

Once again, goes with the attributes.  One other nice thing that I have 
done is the current display class (now called ClassicDisplayView) 
extends from what I called XMLDisplayView and thus it uses an XML back 
end.  This XML backend is a tree structure, so no need to worry about 
efficiency of parsing it.  It is only translated to text at the end.  
Regardless, adding in support for more properties is just a matter of 
looking for them.

> E. Save all status changes (e.g. away messages for IRC).

Yes, we need to track more status changes.  Shouldn't be too hard.

> F. Save file transfer requests.

Hmm.... Interesting idea.  Just another event type.

> G. Save "You have been disconnected" rather than "metoo is offline".
>

Hmm...  Another good idea.

> Is there any way to display the attributes? (e.g. so you can read 
> dates in a web browser)
>

Yeah, you can look at the CSS file I created:
http://fire.sourceforge.net/logs/simple_display.css
There are two sections that are commented out that display the dates, 
but the issue with them is the date is really long.  Even at 8px, they 
are bigger than the message for my example.  You can copy this CSS, 
uncomment those lines, and change the URL for the CSS in your example 
to see what I mean.  I figured it would be more useful to just see the 
messages, and then you can look at the source on the rare occasion 
where you need the date.

> This has been a little more ranty than I intended. Oops.
>
> Perhaps I should subscribe to fire-dev.
>

Yeah, this would have not been a surprise then ;)  We started talking 
about this on the 24th.

> - Purr