Re: Asynchronous xmldb:store()?

Dear All,

For me it is also a problem that updates can take so long, both storing and
updating files. I'd say that a primary limitation of exist as an enterprise
platform is that updating a single node in a large file is way too slow.
So if anybody has any tips on speeding up stores please respond to the
thread.

Having said that, where appropriate I use this function to make
asynchronous calls using the scheduler. It seems like a workaround but it
works ok. Perhaps the exist team should wrap
scheduler:schedule-xquery-periodic-job() in a function called
scheduler:run-async() as an easy way to offer async natively?

declare function local:async-script($script-name as xs:string, $parameters
as element(parameters)?){

    (: Clear job if completed :)
    let $clear-complete-job :=
        if(scheduler:get-scheduled-jobs()//scheduler:job[@name eq
$script-name][scheduler:trigger/state/text() eq 'COMPLETE']) then
            scheduler:delete-scheduled-job($script-name)
        else ()

    (: Only schedule if not already there :)
    where not(scheduler:get-scheduled-jobs()//scheduler:job[@name eq
$script-name])
    return (
        (: Log so we can monitor :)
        util:log('info', concat('async-script:', $script-name)),
        (: Schedule a one-off job :)
        scheduler:schedule-xquery-periodic-job(
            concat('/db/apps/scripts/', $script-name, '.xq'),
            10000,
            $script-name,
            $parameters,
            5000,
            0
        )
    )
};

With best wishes,
Dominic



On Thu, 15 Sept 2022 at 05:32, Lizzi, Vincent <
[email protected]> wrote:

> I have what seems like a very similar question: is there any way to make
> xmldb:store faster?
>
>
>
> I’ve been observing, in a long-running process that stores thousands of
> documents at a time, is that as the database grows in size the time that it
> takes to store documents increases.
>
>
>
> I thought about removing the collection index configuration temporarily to
> store documents first and then run reindex as a second step. However, the
> reindex process can take days on a large collection and there are no
> progress indicators to show whether the reindex is working or hung, so this
> doesn’t seem like a good option.
>
>
>
> Are there any suggestions on what could be done to make xmldb:store faster
> or asynchronous?
>
>
>
> Thanks,
>
> Vincent
>
>
>
> _____________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> [email protected]
>
>
>
>
>
>
>
> Information Classification: General
>
> *From:* Len Schultz <[email protected]>
> *Sent:* Wednesday, September 14, 2022 4:20 AM
> *To:* Craig Berry via Exist-open <[email protected]>
> *Subject:* [Exist-open] Asynchronous xmldb:store()?
>
>
>
> Is there a way to have an XQuery script execute a xmldb:store(), then
> return data without waiting for the store to complete?  My queries can
> create rather large XML variables that I’d like to save for debugging,
> logging, and later inspection.  The stores can take quite a bit of time.
> So I’d like to avoid the performance hit of storing the document on the
> query latency, and therefore the application latency.  Any suggestions for
> how to implement an asynchronous store?
>
>
>
> --len
>
>
> _______________________________________________
> Exist-open mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

_______________________________________________
Exist-open mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/exist-open