Re: Organizing large collection

--===============6568773819968688912==
Content-Type: multipart/alternative; boundary="00000000000022647b0635bea3a3"

--00000000000022647b0635bea3a3
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

If you need to search all the documents each time, then it is most
performant to have them in a single collection to avoid locking overhead
between collections. If you only need to search a subset each time, and you
can divide them by collection and just search one (or a few collections)
this could be faster, but you would need to benchmark based on your own
corpus and search requirements.

Adam Retter

eXist Core Developer
{ United Kingdom }
[email protected]

On Mon, 19 May 2025, 10:23 Alberto Sim=C3=B5es, <[email protected]> wrote=
:

> Hello
>
> Are there differences in terms of performance between having a large
> collection (150k docs) with or without a folder structure?
>
> I want to treat them as a single collection, but I don't know if it helps
> to have sub-collections to organise them, or if that is irrelevant to eXi=
st.
>
> I appreciate any help you can provide.
> Alberto
>
> --
> Alberto Sim=C3=B5es
> _______________________________________________
> Exist-open mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

--00000000000022647b0635bea3a3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto"><div>If you need to search all the documents each time, t=
hen it is most performant to have them in a single collection to avoid lock=
ing overhead between collections. If you only need to search a subset each =
time, and you can divide them by collection and just search one (or a few c=
ollections) this could be faster, but you would need to benchmark based on =
your own corpus and search requirements.</div><div><br></div><div data-smar=
tmail=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div>Adam =
Retter<br><br>eXist Core Developer<br>{ United Kingdom }<br><a href=3D"mail=
to:[email protected]" target=3D"_blank">[email protected]</a><br></div></di=
v></div></div></div></div><br><div class=3D"gmail_quote gmail_quote_contain=
er"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, 19 May 2025, 10:23 Albert=
o Sim=C3=B5es, &lt;<a href=3D"mailto:[email protected]">[email protected]=
om</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margi=
n:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">=
<div>Hello</div><div><br></div><div>Are there differences in terms of perfo=
rmance between having a large collection (150k docs) with or without a fold=
er structure?</div><div><br></div><div>I want to treat them as a single col=
lection, but I don&#39;t know if it helps to have sub-collections to organi=
se them, or if that is irrelevant to eXist.</div><div><br></div><div>I appr=
eciate any help you can provide.</div><div>Alberto</div><br><span class=3D"=
gmail_signature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signa=
ture" data-smartmail=3D"gmail_signature"><div dir=3D"ltr">Alberto Sim=C3=B5=
es<br></div></div></div>
_______________________________________________<br>
Exist-open mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank" rel=
=3D"noreferrer">[email protected]</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/exist-open" rel=3D"=
noreferrer noreferrer" target=3D"_blank">https://lists.sourceforge.net/list=
s/listinfo/exist-open</a><br>
</blockquote></div>

--00000000000022647b0635bea3a3--


--===============6568773819968688912==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline


--===============6568773819968688912==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Exist-open mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/exist-open

--===============6568773819968688912==--