Re: Ideas and Guidance

Hi!

> 1. Let's say I want to grab xml versions of my posts out of the db - 
> what is the easiest way to do that? I browsed the docTool for clues, 
> looking for something like getPostAsRSS or something, but didn't see 
> anything. What am I missing? Or do I have to build my own xml (rss) 
> document from the dict I get from the python calls?

The last choice. :-)

PyDS internally uses dicts and lists and stuff like that. RSS or other
XML formats are just representation. Representation is only created when
something is rendered, not before. There is no need for XPath or stuff
like that, as the stuff already is in a database in fully structured
form.

In PyDS the querying is actually done in two stages: metakit querying is
used to preselect data to reduce the result set to work on, but there is
allways a layer of Python code above that to massage the result set into
the form you need. Tools never directly touch metakit databases (except
their own database, of course) but allways go through exported methods
of other tools (exported methods are methods that don't start with a _).

This has the nice effect that you make the same calls regardless of
where you make them: the same call is used in your internal code, in
your templates (via cheetah) and in external tools (via XMLRPC or SOAP).

> 2. Have you ever considered alternative backend stores for PyDS? I'm
not 
> really serious about this option (at least, not doing this work
myself), 
> but just curious. My thought was, if you could use dbxml as the store, 
> I'd be all set.

No, there are no plans on changing the backend. Sure, it would be
doable, but if I would ever do that, I would go for Python DBAPI, so
people could use different SQL databases. dbxml wouldn't gain me much,
as the data is mostly structured as tables (and subtables), since that's
what Metakit provides (ok, they call it differently, but it's plain old
table stuff).

Metakit was used just because the Python Community Server uses it and I
thought it's a nice and small database. Now we have it, now we use it
;-)

> 3. As a workaround, I thought I could catch change events in the pyds 
> database, extract the info as xml and post it into a separate xml
store. 
> Any pointers on doing this? The 'react to change events' part is
related 
> to my Ping Macro question below.

The easiest way to push stuff into a different database for use by other
tools would be the mirror machinery. You might want to look into CVS in
the ArchiveTool (or use the versions I mailed to this list some weeks
ago). The mirror machinery allows to replicate content to other places.
It's currently used for populating the PyCS search engine and for the
archive database that is used to render archive pages that span multiple
tools.

> 4. Or maybe I should just interact with the metakit db directly? I've 
> started poking around there, which prompted this post - I see many 
> possible ways to get what I want, just wondering if I'm missing an 
> obvious approach or if someone's already considered this question and 
> various approaches already.

This would be a different way. You can pull allmost all data from the
databases by using the PyDS APIs. All PyDS exported APIs are available
through SOAP or XMLRPC, so your tools can work outside PyDS. If you want
to access the metakit files directly, you would need to stop PyDS,
though, as multiple access to Metakit databases isn't a good idea (read
this as "data corruption ahead"!).

Depends on what you want to do and whom you want to give those access
methods - if it's for yourself, you might write your own PyDS tool that
can make direct use of PyDS API (even without XMLRPC or other RPC stuff
- just directly call API methods) and use the interface tools of PyDS to
build your user interface.

If you want to have external tool integration, you should opt for XMLRPC
or SOAP.

If you want to give others access, you would need to do work both in
PyDS (where you would need to render or mirror your data) and the
community server (where you would give others access through special
Community Server modules to the rendered or replicated data).

Yep, I know, choices over choices. :-)

Wether you use rendering or mirroring depends on what you want to do
with the data and how the data looks you want to use. Mirroring is nice
if you have a defined API to push your data into your result database
and if that API is available for PyDS. Rendering is good if you want to
produce some flatfile format from your data, like producing XML.

> to work for me. My last working theory was that it might somehow be 
> related to the combination of being on windows and using ftp 
> upstreaming.
> 
> (Maybe 0.7 fixes it.)

Is (hopefully) fixed in 0.7 :-)

And yes, it was because FTP upstreaming and Windows. There were broken
pathnames in a place where they were problematic. I couldn't check this,
though, as I don't have Windows.

> Now I'd like to have it run whenever I post automatically. In general, 
> what's the way to do this, ie, to have a custom macro run whenever a
new 
> post is made or edited? Do I want to think about a tool for this?

That's what PingTool is written for, so maybe it's best to just use that
one ;-)

If you want to do things when posts are changed, again the mirror
machinery is the right thing to look into. The mirror machinery is
spread over MirrorRegistry.py, MirrorTool.py and the mirror
implementations (PyCSSearchMirror is in 0.7, ArchiveTool is in CVS).
Mirroring is triggered as soon as postings are changed, created or
deleted. Currently weblog posts, blogmarks, stories (both normal and
external stories) and wiki nodes are mirrored.

PingTool works differently: it hooks into the UpstreamTool. So whenever
a /rss.xml URL is upstreamed, the pings are triggered. That was the
reason for your problem: the hook actually didn't see a /rss.xml file,
but a \rss.xml file and so the hook didn't trigger.

The problem with upstream hooking is that you can only hook on
filenames. And that's where the problem begins: files and elements are
not 1:1, necessarily. Think of the weblog, where one posting can be in
several files and where one file can contain several postings.

Mirroring directly works on posting level.

> special things for me on the desktop. It is written in python. I don't 
> know too much about its internals (yet). What I'd like to do (I think) 
> is essentially merge fooDS into PyDS so that I have one desktop server 
> that functions as a 'hub' for me.

This largely depends on what fooDS does. Usually it shouldn't be too
problematic to rewrite some server into a PyDS tool. A PyDS tool
provides most of the stuff a standalone server would need, too. PyDS
tools have a way to build web based user interfaces, you have the
internal tool methods and you have a database for stuff you need to
keep. And then you have a background thread (if you choose to have one)
that sit's there and waits for work that takes much time and should be
done in the background. So PyDS.Tool is more like a framework for
building servers that are integrated into a master server. The nice
thing is that a tool can make use of all functionality of other tools,
so you have the rendering engine and other nice stuff handy if you need
it.

There will be some changes in PyDS internal methods for tool
construction and registration with 0.7.1 (the new threading model makes
this necessary), so if you do some work on tools, be prepared for some
needed rewriting (mostly very simple changes, nothing big - the big
stuff is in the internals of PyDS).

> functionality? What other extensibility mechanisms for server
capability 
> is there?

You could hack into medusa and add your own handlers if your needs are
more special than a PyDS tool can handle. But if you do that, you are on
your own with handling of longrunning tasks, so it might be a good idea
to not do it :-)

I wrote a bit about Medusa, threading and longrunning computations just
recently on my blog.

> Sorry for such a long post, it didn't start that way.

No problem. Hopefully someone collects all this stuff and writes a
documentation ;-)

bye, Georg