Re: eXist-db repair and reindex process taking a very long time

Hi Vincent,

when I’m sure that no write has taken place, I go along the same route as Pieter Lamers said in his post, i.e. deleting the log and all lock files. eXist then does not notice there has been an interruption and it should come up without re-index.

As I, too, have a lot of files and a large index definition, I usually try to avoid re-indexing. Meaning, if I know it’s going to re-index, I delete the whole thing and just do a re-ingest of all files from a recent DB dump. 1) you can go through your files collection by collection instead of doing everything at once (which gives you a much better overview of where in the process you actually are); and 2) the DB is not completely locked (which it is during the automatic re-index).
In my experience, the re-ingest is not significantly slower* than the automatic re-index and to me the benefit of knowing where you are in the process is worth more than a small speed bonus.

* While I have not timed it, it feels than a re-ingest is faster than the re-index. But I do not know nearly enough about eXist’s internal handling of re-indexing and indexing when ingesting to say whether there’s really a difference or other effects play into this.
Of course, when re-ingesting from a dump and that dump is situated on a different piece of hardware, a re-ingest would also benefit from the separation of the I/O tasks which might give some added boost (again, I have not really timed this in a clean environment).

--

That being said, almost 1 month of re-indexing and 100GB mem usage seem quite a lot. I do not dare say “excessive“ as I do not know your data and index definition, but my data’s usually done after about 3 to 4 days (no dedicated hardware and a max. of 6 GB RAM – a re-ingest is around 2–3 days, sometimes less).

How big are you data\lucene and data\range directories and how big are the dbx files? That might help compare these figures.

All best,
Dario


Am Dienstag, dem 10.09.2024 um 19:10 +0000 schrieb Lizzi, Vincent:
Hello eXist-db community,

By comparing the folders data\lucene and data\range to a previous backup, going by the number of files and total size of these folders, it looks like the reindexing process is about 50% done. The reindexing process has been running since August 13 and is using over 100 Gb of memory.

Is there any way to start eXist-db and allow it to go through its recovery process but stop it from reindexing database files?  I’m wondering if that could be a way to get the database operational again, and then I could manually run xmldb:reindex().

Thanks,
Vincent

_____________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
[email protected]<mailto:[email protected]>



Information Classification: General
From: Lizzi, Vincent
Sent: Tuesday, September 3, 2024 11:36 AM
To: Exist-open <[email protected]>
Subject: eXist-db repair and reindex process taking a very long time

Hello eXist-db community,

I’ve been monitoring an eXist-db database that is going through its automated recovery process, and am wondering if there is any way to get more information about its progress and how soon the process will finish.

This eXist-db has full text indexing and range indexes configured on several large collections. The EC2 server on which eXist-db is hosted had an outage. When eXist-db was restarted its automatic repair process began. That was about 3 weeks ago. Through Windows Resource Monitor I can see that the eXist-db process is reading from dob.dbx and structure.dbx and writing to structure.dbx and writing to files in the “lucene” and “range” folders, and the process is using about 6% of CPU consistently, and memory usage has increased gradually to over 100 GB. The last line in exist.log is still:

2024-08-13 17:02:59,710 [main] INFO  (NativeBroker.java [repair]:3692) - Reindexing database files ...

Is there any way to find out more what the process is doing, estimate when it will finish, or release any bottlenecks, without interrupting the process?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail:[email protected]<mailto:[email protected]>
Web:www.tandfonline.com<http://www.tandfonline.com>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."

_______________________________________________
Exist-open mailing list
[email protected]<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/exist-open


--

Dario Kampkaspar
Leitung ZEiD – Zentrum für digitale Editionen
Universitäts- und Landesbibliothek Darmstadt

Postadresse:
Magdalenenstr. 8
64289 Darmstadt

Besucheradresse:
Residenzschloss 1
64283 Darmstadt

+49 6151 16-76292
+49 151 29121599

_______________________________________________
Exist-open mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/exist-open