MF Media Tools project

MFers,

I've sketched out very preliminary ideas about the MF media tools project
that Bij and Dru and I have been chatting about a little bit.

The initial doc is included here, but I'll be updating it at

  http://monkeyfist.com/kendall/media-tools.txt

so the latest-greatest will be there for a while.

Any help, including hacking, will be greatly appreciated.

Kendall

------------------------------
Monkeyfist Media Tools Project
------------------------------

The idea is to create a site and tools to help MF (and others) do
corporate media analysis. Eventually I'd like the site to be aliased
as "media-tools.indymedia.org" for use by the whole IMC movement.

We'll use the site to analyze corpo media coverage of events and
issues that Monkeyfist and IMC covers; i.e., facilitating tools to aid
leftist media analysis. (Eventually expanding to include analysis of
FOIA requested documents of all sorts; like the 5,000 pages of Seattle
protest FOIA stuff I'm getting, which includes 1,000 pages of corpo
media reports.)

The conceptual foundation of the site and the tools is Herman and
Chomsky's Propaganda Model (and I suspect the site will include essays
about understanding and applying the Model -- very vital skills for
indy media folks).

Tools We Want
-------------

 * Archive -- first step is to identify relevant URIs and archive
   their content
 
 * Search -- full-text search of the archive (Swish, Glimpse, plus the
   possibility of a custom search tool)

 * Track -- browse coverage by 1) issue, 2) event, 3) publication,
   4) author, 5) chronology, 6) (other metadata)

 * Memewatch -- One of the conceptual claims behind this project is
   that, given the ideological and economic structure of corpo media,
   there are a relatively small number of claims that *can* be made
   about any particular issue, for example, antiglobalization. This
   means tracking the continuities and the changes can be a powerful
   tool. For the antiglobalization event coverage, there are around 30
   cliches that the corpo media continually recycles; this tool would
   track the presence of those cliches in corpo media stories. (So
   basically this comes down to something like a table of cliches, and
   a metadata field on each item that lists which cliches are present
   in the story (which we may demarcate by markup).)

   (Actually, I've been a bit sloppy here; we want to track two kinds
   of thing: memes (repeated ideas, which may or may not be right,
   true, or fair); cliches (memes which are wrong, false, or unfair,
   and which form part of the prevailing conventional wisdom); from a
   technical point of view, these are the same thing, but need to stay
   separated since sometimes we'll want to know about cliches,
   sometimes about memes. There will be overlap between the two
   things; that is, sometimes a thing is both a cliche and a
   meme.)

 * Memespread -- This is a cross between a chronological view of a
   subset of the archive and a "meme" view; that is, tracking the
   first and earliest occurrences of a "meme", how use of the meme
   spreads, etc. (This concern comes from two sources: Bijan is really
   into this, but it's also predicted by the Propaganda Model; the
   economic pressures on corpo media suggest that most media outlets
   outsource as much reportage as possible; practically speaking, this
   means that many reports rely on the AP and Reuters wires; and
   almost all editorialists rely on these reports, which rely on the
   wire services.) Being able to track the rise and spread of a meme
   -- "Carlo Giuliani had a troubled past, including trouble with the
   police" -- can be a very useful thing to have to debunk nonsense.
  
   I think, though haven't thought about it very much, that Memespread
   tool may emerge from developing the rest of the system smartly; it
   seems to require 1) normalized "time of publication" (indexed to
   UTC, I suppose) data for *every* item added to archive; 2) a clever
   data model for cataloging and normalizing (this will just be
   editor/person-power intensive to prevent "semantic drift"; not
   terribly hard, just has to be done by a person) memes (which is
   bound to be very dynamic, given how this data tends to act "in the
   wild"), and associating occurrences of memes with resources (which
   is a many to one relation; every resource can be an instance of
   zero, one, or more memes; every meme is instanced by one or more
   resources (I think -- memes without instances are something else
   entirely; though they are predictable so there may be some sense in
   being able to say to the system, "We need to start a "Carlo is
   violent" meme since he was just killed and the media will demonize
   him now); 3) a way to associate archived resources with one or more
   issues and one or more events. If we have these three things, it
   should be possible to build queries like: "Show me all the articles
   where EVENT == Genoa, Meme == [1, 3, 17], and SORT = reverse
   chronology" -- I'm clearly no SQL guy, but that seems *roughly*
   right.

 * Talking Points -- High-quality responses to memes, which people can use
   in their activism.

 * Pull Quotes -- We need to be able to say of a resource that there are
   zero, one, or more "notable quotes" and here they are. (A huge boon to
   indy media editorialists.) Initial thought is to mark quotes in-place
   with XML; but it may make more sense to store them separately. Unclear.

 * UI/Admin for all of this (a Dru FIPesque by-paragraph/sentence commentary
   UI might be helpful in a few places. Would let us do stuff like, "This
   sentence is cliche 7 and meme 24" and "This paragraph is a pull-quote".)

 * A metadata scheme -- I'll steal from a combo of NITF and NewsML, plus
   Dublin Core; the richer the metadata we keep, the better queries should
   be possible. (Or something like that.)

 * an RSS model to syndicate our stuff; very important to make our data as
   widely available as possible; prolly also an XML-RPC/SOAP interface to
   let trusted sites and clients connect to the server and get back useful
   results. Be kind of cool if every IMC/indy media site had "Memewatch"
   headlines; this might be a form of anti-corpo-media hacktivism; making
   it harder for them to spread their crap.

A possible design pattern: store archived resources as text (XML,
plain text, HTML) on disk, use Swish or Glimpse or Locus for plain
text searches; every resource gets an entry in Postgres, including
metadata and key to lookup the full-text in the filesystem (that is,
something to steal from Gavin at Driveway; store stuff in the FS but
do so in an FS-independent way so that the database knows as little
about the way files exist on disk as possible); we do queries against
the database and full-text searches against the "pile of files" on a
fast, ReiserFS SCSI disk.

I'd like to use Python Webware, AOLserver, and Postgres for this project.
(Which lets us integrate it with some parts of OpenACS, some of which would
be very useful; forums, notifications, etc.) Alernately, Webware and Apache.
(I'm going to have an AOLserver box up soon so that's not a barrier; and it
is fast as hell.)

Kendall Clark (stealing ideas from Bijan Parsia, Dru Jay, and others)
$Id: $