pseudo blog 17-May-2004

so far this spring, three people that have had a major influence on my=20
life have passed away.  I've already spoken about Nick.  The next was=20
Janet Mattie, head of the AAVSO (American association of variable Star=20
observers).  She and her surviving husband were both very influential in=20
my interest in astronomy.  If it hadn't been for the fact that the=20
astronomy field was generating 500 Ph.D.'s for every 50 openings, I=20
might have stayed there.  Janet did many things for me including=20
developing an appreciation of why it is necessary to do good data=20
gathering and why it is so very important even  with tools like Hubbell.=20
  This appreciation has served me well even in the domain of software=20
development.

the next person was my father-in-law.  Like all humans he had feet of=20
clay but he did allow me to live in his house for a couple of weeks when=20
I left home to start my own life.  He helped me with finding my first=20
job and the encouragement to continue in the software development field.=20
  Decidedly a mixed blessing.

we have a yearbook from the time when he served on the USS Avery Island.=20
  He was part of the bikini nuclear weapons tests and was only 15 miles=20
from ground zero.  The plan is to scan the yearbook and publish it on=20
the net.  I pity my poor little DSL line.

and that's my excuse for being slow...
-------------

camram notes.  Two important developments: multi domain/server front-end=20
instance and Windows client side stamper.

The first development is that I now have a version of camram that will=20
work as a "lump in the line".  You can install it in front of any=20
mailserver, handle any number of domains, and it can handle all traffic=20
from the Internet and from your mailserver doing the right thing=20
(stamping, filtering etc.).  It's still, as they say, dammed fragile but=20
it should be ready for testing relatively soon.  If you are interested,=20
please e-mail me directly so we can work out the details.

I must say that the reason it took so long to develop was because I went=20
down a rathole of using milter as the filter interface to sendmail.=20
fortunately, the process revealed (albeit gradually) some of the=20
architectural changes I needed to make in camram.  Some of the more=20
painful and there are still some problems that will impede=20
multithreading through the filter but that's for another day.

The current system uses two copies of e-mail relay and one copy of=20
sendmail.  The first copy of e-mail relay takes traffic in from the=20
Internet side of the net and filters it.  The messages then relayed to a=20
delivery only instance of sendmail which has all the routing information=20
necessary for proper handling of e-mail.  The second e-mail relay=20
instance is set up exactly the same way except it is excepting e-mail=20
from the "internal" side of the network and only performs stamping=20
operations.  Again when it is done, the e-mail is passed to the delivery=20
sendmail instance.  If the server is exposed directed for the Internet,=20
you can also put two more instances of sendmail in front of the e-mail=20
relay proxies.  The reason for this is to allow you to add something=20
like smtpauth for the internal interface and to give you greater traffic=20
handling capability with queueing for your Internet sourced traffic.

Yes, there is a question about performance but I believe that it=20
shouldn't be too horrible and that camram itself will be the limiting=20
factor, not the architecture.  In practice with the procmail delivery=20
model which is effectively the same thing as what I've described here,=20
the dominant overhead is crm114.  When the system has learned who you=20
speak with, the overhead is much lower.

another point is that there's nothing special about sendmail in this=20
context.  a full MTA for queuing, message handling, deliveries etc.,=20
anything will work as long as you can fully control source and=20
destination interface bindings and port members.

for an intermediary tool for interfacing with camram, E-mail relay has=20
been a far more friendly and flexible tool for these purposes.  I cannot=20
say enough good things about it.  Only thing that would make it better=20
would be if Python was better integrated (hint, hint).  I suspect that=20
if anybody wants to help Graham with that, it would be welcome.

The second development is that I now have a version of the stamp or=20
working on Windows.  It's only slightly less fragile than the lumper=20
version of camram but it does work and, in fact I am using it right now.

it's more of a proof of concept than a real implementation but it gives=20
us a foundation for the user experience generating local stamps.

implementation is two email-relay instances.  First for accepting=20
traffic and queuing up the messages.  The second is for stamping then=20
delivery.  I will admit that it has been a bit of a hair tear.  The=20
author of e-mail relay Graeme Walker has been wonderfully supportive.

On the window side of the house, the interface for calling the filter=20
only can execute executables or batch files and I had to modify (poorly)=20
a JavaScript wrapper to call the Python instance which would run my=20
Python code.  I believe this is known the Russian dolls school of code=20
development.  the same set of components applied to MacOS X should also=20
yield a MacOS X client side Stamper.

theoretically too I also have it set up to autostart.  I will test=20
sometime in the near future unless one of you want to beat me to it.

If you want a copy, let me know and I will send you the super secret URL.

I must admit, this combination of bits has some interesting possibilities=
.

The last thing I'm working on is user interfaces for the training and=20
dumpster sections.  I found a bug in the process I was using for=20
training CRM114.  I was apparently overtraining and getting distorted=20
score patterns.  I'm now doing strictly what is known as "train on=20
error" and things seem to be working better.  The discrimination seems=20
good enough that you can use a very narrow set of limits and really=20
minimize the amount of work necessary for training.

I will start checking things into CVS soon and generate up a release one=20
can use with the raging dormouse acquisition package.

---eric