Breaking the core with CORTEX

I've just finished tweaking this system, and am starting to do stuff 
with it, so I thought I'd write it up for those who may find it useful 
(primarily core and UI developers). This document will go on the 
website if it's at all useful - please comment on it!


							----------

						Using CORTEX

CORTEX is the Everybuddy core test suite. It is used to automate testing 
of the core by batch-executing GUIcomms commands. It is primarily 
useful to core developers, for stress-testing and debugging the core, 
and verifying that your latest change hasn't broken something vital. It 
is also a good tool for bug reporting - a cortex script which generates 
an error is the best possible way of duplicating a bug - it means that 
other developers can generate the same bug reliably, making it that 
much easier to fix.

A cortex script consists of a sequence of tests. A test looks something 
like this:

{SEND, {"message_hold", "0"}, 2},

C programmers may recognise this syntax - in fact, a cortex script just 
defines an array of C structures which are #included when compiling the 
program. The first field is a "type" - SEND (send the given GUIcomms 
command), EXPECT (read a command and complain if it's not identical to 
the one given), WAITFOR (skip all commands until one arrives matching 
the one given), MESSAGE (print a message to the console), and END (end 
the script - this must be given once, as the last element of the array, 
or cortex will die messily).

For example, the following script tests that the AIM module is loaded 
and reporting itself correctly:

{
{EXPECT, {"cookie_accepted"}, 1},
{EXPECT, {""}, 0},
// ^^^ a "command length" of zero means "accept any command"
// This line discards the potentially variable away message.
{EXPECT, {"message_hold", "1"}, 2},

{MESSAGE, {"Checking for AIM service"}, 1},

{SEND, {"list_services"}, 1},
{WAITFOR, {"list_service", "AIM", "#000088", "6"}, 4},
// ^^^ Use of WAITFOR makes cortex skip the responses for any services
// listed before AIM
{EXPECT, {"list_service_actions", "AIM", "buddy", "0"}, 4},
{EXPECT, {"list_service_actions", "AIM", "groupchat", "0"}, 4},
{EXPECT, {"list_service_actions", "AIM", "group_user", "0"}, 4},
{EXPECT, {"list_service_states", "AIM", "3", "Online", "Away",
	"Offline"}, 6},
{EXPECT, {"list_service_done"}, 1},
{WAITFOR, {"list_services_done"}, 1},

{END}
}


Now, as it stands, this is really very laborious to type out. Perhaps 
usable for an automated pre-release test, if any human could stand the 
drudgery of typing it all out. However, there's no need for a human to 
do this. The latest version of eb-console now supports the -t option, 
which puts a transcript of your session, in the format of a cortex 
script, into the specified file. This is rarely useful as an entity 
(barring its use to report bugs, of course), but from here it becomes 
trivial to copy and paste lines from a transcript into your cortex 
script. This is how I am creating the pre-release tests I will be using 
from alpha-11 onwards.

So, to summarise:

(*) CORTEX is a program for testing the EB core and reproducing bugs
(*) It works by "playing back" sequences of GUIcomms commands, and 
complaining if the core deviates from the script
(*) eb-console can be used to record transcripts to be played back later 
(or by someone else)

What do you think? Can you see yourself using it? Is that document clear 
enough?

Meredydd

PS:
As regards program availablility - I'm still stuck behind this firewall, 
but I'm going back home for a while on Friday (the 13th, gulp), and 
will commit the latest versions of cortex and eb-console then. Until 
then, to use the examples here, you'll need to get them from:
http://www.everybuddy.com/~meredydd/cortex.tgz
http://www.everybuddy.com/~meredydd/eb-console.tgz