RE: Harvesting infrastrucure (was: Don't kill Bert's workstation)

> 
> Please remember these are not UIDs, just a count from 1 to 
> number-of-messages-in-archive. If something (someone?) (me??) 
> messes up 
> my mbox file, these numbers will change.
> 

Right, I know they're not actually UIDs.  And in the scenario I propose,
I guess they *still* won't be UIDs -- the process will parse and export
individual email .txt files, numbered in the order they're extracted.
But there is no safeguard against a hiccup in the parsing process.
Ideally someone should be able to download the mbox file from
lists.squeakfoundation.org, parse it, and get the same output as my
(theoretical) server.

Anybody got suggestions for a good algorithm generating the UID?  How
about a hash just using the subject line and the date? 


> Anyway, I've added this:
> 
>	http://swiki.gsug.org/sqfixes/last
>
> which gives you the number of the last message as plain text (same for

> sqbugs). I don't see the point in generating a list of descending 
> numbers starting at 3375 - you can easily count yourself ;-)

Very cool!  Thanks Bert.  This is exactly what I want.  It will speed up
the "load updates" process significantly.
From [email protected] Thu Jun 05 17:59:45 2003
Return-Path: <[email protected]>
Delivered-To: [email protected]
Received: (qmail 11988 invoked from network); 5 Jun 2003 17:59:45 -0000
Received: from unknown (HELO lucy.riskmetrics.com) (12.3.62.14)
  by mail.theinternetone.net with SMTP; 5 Jun 2003 17:59:45 -0000
Received: from riskmetrics.com (w018.z064003232.det-mi.dsl.cnc.net
	[64.3.232.18]) by lucy.riskmetrics.com with SMTP (Microsoft Exchange Internet
	Mail Service Version 5.5.2653.13)
	id MGR10HF1; Thu, 5 Jun 2003 13:58:46 -0400
Message-ID: <[email protected]>
Date: Thu, 05 Jun 2003 13:58:21 -0400
From: Doug Way <[email protected]>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
	rv:1.3.1) Gecko/20030425
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Discussing the Squeak Foundation
	<[email protected]>
Subject: Re: [Squeakfoundation] Harvesting infrastrucure
References: <[email protected]>
In-Reply-To: <[email protected]>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: [email protected]
X-Mailman-Version: 2.1
Precedence: list
Reply-To: Discussing the Squeak Foundation
	<[email protected]>
List-Id: Discussing the Squeak Foundation
 <squeakfoundation.lists.squeakfoundation.org>
List-Unsubscribe: <http://lists.squeakfoundation.org/listinfo/squeakfoundation>,
	<mailto:[email protected]?subject=unsubscribe>
List-Archive: <http://lists.squeakfoundation.org/pipermail/squeakfoundation>
List-Post: <mailto:[email protected]>
List-Help: <mailto:[email protected]?subject=help>
List-Subscribe: <http://lists.squeakfoundation.org/listinfo/squeakfoundation>,
	<mailto:[email protected]?subject=subscribe>
X-List-Received-Date: Thu, 05 Jun 2003 17:59:46 -0000

Bert Freudenberg wrote:

> (moved to foundation list because ... it belongs here)
>
> Am Donnerstag, 05.06.03 um 16:16 Uhr schrieb Brent Vukmer:
>
>> Cool.  Would you be willing to eventually setup another "Bug Fixes
>> Archive Server" image, running on your box in parallel with the current
>> PWS-MailArchiveAction image and accessible from, say,
>> http://swiki.gsug.org/bugfixesarchive ?
>>
>> The BugFixArchive-Model framework has the beginnings of an API to
>> replace the PWS-MailArchive stuff -- at least from a back-end
>> perspective.  Right now the 1.1 API provides an email-file-repository
>> directory with static email .txt files ( and subdirectories with
>> extracted attachments ).  You could configure your Apache to just serve
>> up those static files without hitting Squeak at all.
>
>
> In principle, that is no problem. In practice, I probably will leave 
> this place later this year, at which point this machine will go away, 
> too. Swiki.gsug.org will be moved to another box, but I can not 
> guarantee yet that the sqfixes archive will continue to work in its 
> current form.
>
> So, in any case, it would be better if you implement this solution on 
> the squeakfoundation server. 


I agree that we probably shouldn't rely on Bert's server long-term.  
Brent's UI is quite nice, though, and maybe we could switch the back-end 
to something else at some point.  (Could be based on SqueakMap, 
Monticello, or just a bunch of files on a server.)  Unfortunately I'm 
not strongly motivated to work on this right away since I have plenty of 
other stuff to do, and what we have seems to work pretty well. ;-)  
Something to keep in mind, though.

- Doug Way