Re: high memory usage when constructing a string

Vincent,

As Jean-Paul said in his reply, there are alternative ways to build large outputs.

Maybe describe the outcome you are trying to achieve as opposed to the issue you are finding. There are generally more than one way to do things in eXist.

Alasdair

Sent from my iPhone

On 11 Aug 2023, at 1:42 am, Lizzi, Vincent <[email protected]> wrote:



Hi JPR,

That sounds like a good idea. Unfortunately, trying to use JSON for this amount of data also results in out of memory errors.

array:join(

(1 to 200000000) ! ['1']

) => serialize(map{'method': 'json'})

I have gotten eXist to process actual data in tab separated text format when eXist is freshly started and has plenty of memory available, but even in this ideal condition memory usage is still unexpectedly high and sometimes produces out
of memory errors.

Thanks,

Vincent

_____________________________________________

Vincent M. Lizzi

Head of Information Standards | Taylor & Francis Group

[email protected]

Information Classification: General

From: Jean-Paul Rehr <[email protected]>

Sent: Thursday, August 10, 2023 10:06 AM

To: Exist-open <[email protected]>

Subject: Re: [Exist-open] high memory usage when constructing a string

Dear Vincent,

Have you tried outputting JSON structure data from the database? It will be easier to store and access later using Xquery via arrays and maps. You could even chunk it if eXist chokes on a large file - I've never handled one of that size.

Cheers,

JPR

On Thu, Aug 10, 2023 at 3:24 PM Lizzi, Vincent <[email protected] > wrote:

Hello eXist folks,

I’m observing that asking eXist to construct a string of about 200 Mb uses more than 4 Gb of memory.

When running a query such as this that constructs a string of 200,000,000 bytes:

string-join(( (1 to 200000000) ! '1' ))

Monex shows the memory usage climbs to over 4 Gb.

This sometimes exhausts the available memory and results in a stack trace being printed exist.log, the top of which shows:

2023-08-09 23:12:40,553 [qtp353417634-34] ERROR (XQueryServlet.java [process]:549) - Java heap space

java.lang.OutOfMemoryError: Java heap space

at org.exist.dom.memtree.DocumentImpl.addChars(DocumentImpl.java:273) ~[exist-core-6.2.0.jar:6.2.0]

at org.exist.dom.memtree.MemTreeBuilder.characters(MemTreeBuilder.java:382) ~[exist-core-6.2.0.jar:6.2.0]

I would guess that constructing a document uses a lot more memory than constructing a simple string. Is there a way to signal eXist to use a simple string builder instead?

The example above is a simplification of what I actually want to do, which is to retrieve data from a SQL database and store the results in eXist to use for later processing. The
data contains nearly 5 million rows with only 2 columns, so tab separated text would seem to be the most efficient way to store this data in eXist. Here is the relevant portion of the query:

let $data as xs:string :=

string-join(

sql:execute($cn, $sh, (), true())/sql:row/string-join(*/text(), '&#x9;')

, '&#xA;')

return xmldb:store-as-binary('/db/temp', 'list.txt', $data)

Is there a more efficient way to do this in eXist?

Thanks,

Vincent

______________________________________________

Vincent M. Lizzi

Head of Information Standards | Taylor & Francis Group

530 Walnut St., Suite 850, Philadelphia, PA 19106

E-Mail:
[email protected]

Web:
www.tandfonline.com

Taylor & Francis is a trading name of Informa UK Limited,

registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."

Information Classification: General

_______________________________________________

Exist-open mailing list

[email protected]

https://lists.sourceforge.net/lists/listinfo/exist-open

_______________________________________________
Exist-open mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/exist-open

_______________________________________________
Exist-open mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/exist-open