Issues with Composite DataSource & how to replace

Neil Stansbury <neil.stansbury@REPLY_redbacksystems.com> Fri, 20 May 2005 18:07:48 +0100
Newsgroups gmane.comp.mozilla.devel.rdf
Organization Another Netscape Collabra Server User
Message-ID <[email protected]>
I am am having increasingly strange results with a Composite DS, it's 
hard to produce test cases, but most result in either slow or non 
existent aggregation, and perhaps too many separate DS's.

One problem is my remote (HTTP/s) URLs requested with GetDataSource(uri) 
contain query strings ( uri/path?arg=1&arg=2 etc ).

In part the uri arguments are to allow an incremental loading of large 
amounts of RDF data as the CDS doesn't appear to give access to either 
an nsIStreamListener or nsIProgressEventSink.

As a result of this, I can't separate the URI from the arguments and so 
nsIRDFRemoteDs.Refresh(false) makes no sense because of the URI args.

Updating assertions directly in the composite directly rather than 
finding a member one seem to frequently end up with multiple assertions 
rather than aggregates.

Also, I am flushing the data from a CDS to an nsIFile by iterating and 
then serialising each member ds into an nsIFileOutputStream. But not 
only are the namespaces not making sense (things like NS1: and NS2:), 
but obviously the results aren't aggregated.

Anyway, considering all this I am considering moving to an in-mem-ds, 
and either using an nsIHttpChannel with it's notifications, and manually 
parse the data, or use an nsIRDFXMLParser and pass the in-mem-ds into 
the parseAsync() method and get access to it's nsIStreamListener that way.

This way I get one true datasource for all my needs, and not need worry 
about URI arguments and flushing etc.

Can anyone give me any pointers here - particularly on how aggregation 
would work loading from remote HTTP/s URL. From looking so far it 
appears I have access to all the rdfObservers and Sinks I would do 
normally, so the change over should be minimal?

Thanks in advance,

Neil