Re: xindice_rebuild - file size vastly multiplied

--0016e64ea81e276ba10467c192b9
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Apr 16, 2009 at 3:32 PM, OKO <[email protected]> wrote:
[snip]
>
> The "1.1" vs "1.0" expansion factor is much larger for the case of
> collections with few and small files (factor 350) , than for many and sma=
ll
> files (factor 25).
>
> But nevertheless, I wonder for what kind of databases 1.1 would beat 1.0
> from a *size* point of view. Maybe comparing processing speed of "1.1" vs
> "1.0" would give a different picture, but at the moment I have no numbers=
 to
> offer in that respect.

Here is the deal: database format between 1.0 and 1.1 hasn't changed,
except some bugs in index files were fixed and the collection files
are now created properly, which in your case, unfortunately, means
they take a lot more space than before. The reason is that default
size is intended for bigger collections with many documents.

The database you tried to rebuild is the corner case however, with
several collections that only have a single document each, so default
size settings are way too large. But this is only the default setting
and it can be change to more suitable number. This way, 1.1 database
should take about the same amount of space as the existing database.

> By the way, I do not have any very large documents in the database, so I
> have no idea how "1.1" compares to "1.0" for such beasts.
>
> And the 64,000 $ question is, of course, what means are available to
> decrease disk space footprint of the "1.1" =A0database?

The following steps can help to reduce database size:
1. Meta collections can be turned off. The setting is in the
config/system.xml file (In Xindice 1.1 directory). In the following
line:

    <root-collection dbroot=3D"./db/" name=3D"db" use-metadata=3D"on">

change use-metadata=3D"off". This will make all Meta collections go away.

2. Initial collection size can be adjusted (here I assume that all the
collections use default BTreeFiler to store data, HashFiler is
somewhat different beast). When creating a collection it can be given
a configuration to specify pagecount setting that directly affect
initial collection size:

<collection compressed=3D"true" inline-metadata=3D"true" name=3D"test">
  <filer class=3D"org.apache.xindice.core.filer.BTreeFiler" pagecount=3D"16=
" />
</collection>

When creating collection from command-line tool, this setting can be
specified with --pagecount parameter:

bin\xindice ac -c /db -n test --pagecount 16

The "perfect" value for pagecount depends on size and amount of
documents in a collection. For collections that have documents added
and modified often it should be picked based on how much data is
expected to be stored in the collection, but in any case it only sets
initial size, so collection file will grow when it is too small to
hold new data.

In your case, when a collection has only one small document and this
situation is not expected to change, pagecount may very well be set to
1.

Now back to data migration... I made some changes to xindice_rebuild
to add optional pagecount parameter that will overwrite original
collection setting.

If you have source version of Xindice download, please save attached
file to <xindice
directory>\java\src\org\apache\xindice\tools\DatabaseRebuild.java and
run build.bat or build.sh depending on your OS. After that, you can
try to rebuild database again using optional parameter:

bin\xindice_rebuild.bat rebuild db -p 1

If you have binary version of Xindice download instead, please let me
know, I'll see what can be done for that. Alternatively, collections
can be rebuilt manually, by exporting documents, creating new
collections using command-line tool with the option "--pagecount 1"
and importing documents into new collections.

Let me know if something doesn't work for you.

Natalia

--0016e64ea81e276ba10467c192b9
Content-Type: application/octet-stream; name="DatabaseRebuild.java"
Content-Disposition: attachment; filename="DatabaseRebuild.java"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_ftn09hca0

LyoKICogTGljZW5zZWQgdG8gdGhlIEFwYWNoZSBTb2Z0d2FyZSBGb3VuZGF0aW9uIChBU0YpIHVu
ZGVyIG9uZSBvciBtb3JlCiAqIGNvbnRyaWJ1dG9yIGxpY2Vuc2UgYWdyZWVtZW50cy4gIFNlZSB0
aGUgTk9USUNFIGZpbGUgZGlzdHJpYnV0ZWQgd2l0aAogKiB0aGlzIHdvcmsgZm9yIGFkZGl0aW9u
YWwgaW5mb3JtYXRpb24gcmVnYXJkaW5nIGNvcHlyaWdodCBvd25lcnNoaXAuCiAqIFRoZSBBU0Yg
bGljZW5zZXMgdGhpcyBmaWxlIHRvIFlvdSB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNp
b24gMi4wCiAqICh0aGUgIkxpY2Vuc2UiKTsgeW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNl
cHQgaW4gY29tcGxpYW5jZSB3aXRoCiAqIHRoZSBMaWNlbnNlLiAgWW91IG1heSBvYnRhaW4gYSBj
b3B5IG9mIHRoZSBMaWNlbnNlIGF0CiAqCiAqICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGlj
ZW5zZXMvTElDRU5TRS0yLjAKICoKICogVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3
IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQogKiBkaXN0cmlidXRlZCB1bmRlciB0
aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAogKiBXSVRIT1VU
IFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3Ig
aW1wbGllZC4KICogU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292
ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAogKiBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4K
ICoKICogJElkOiBEYXRhYmFzZVJlYnVpbGQuamF2YSA1MzMyNTMgMjAwNy0wNC0yNyAyMzoxMjox
MlogdmdyaXRzZW5rbyAkCiAqLwoKcGFja2FnZSBvcmcuYXBhY2hlLnhpbmRpY2UudG9vbHM7Cgpp
bXBvcnQgb3JnLmFwYWNoZS5jb21tb25zLmxvZ2dpbmcuTG9nOwppbXBvcnQgb3JnLmFwYWNoZS5j
b21tb25zLmxvZ2dpbmcuTG9nRmFjdG9yeTsKaW1wb3J0IG9yZy5hcGFjaGUueGluZGljZS5jb3Jl
LkNvbGxlY3Rpb247CmltcG9ydCBvcmcuYXBhY2hlLnhpbmRpY2UuY29yZS5EQkV4Y2VwdGlvbjsK
aW1wb3J0IG9yZy5hcGFjaGUueGluZGljZS5jb3JlLkRhdGFiYXNlOwppbXBvcnQgb3JnLmFwYWNo
ZS54aW5kaWNlLmNvcmUuZGF0YS5LZXk7CmltcG9ydCBvcmcuYXBhY2hlLnhpbmRpY2UuY29yZS5k
YXRhLlZhbHVlOwppbXBvcnQgb3JnLmFwYWNoZS54aW5kaWNlLmNvcmUuZmlsZXIuQlRyZWVDYWxs
YmFjazsKaW1wb3J0IG9yZy5hcGFjaGUueGluZGljZS5jb3JlLmZpbGVyLkJUcmVlRXhjZXB0aW9u
OwppbXBvcnQgb3JnLmFwYWNoZS54aW5kaWNlLmNvcmUuZmlsZXIuQlRyZWVGaWxlcjsKaW1wb3J0
IG9yZy5hcGFjaGUueGluZGljZS5jb3JlLmZpbGVyLkZpbGVyOwppbXBvcnQgb3JnLmFwYWNoZS54
aW5kaWNlLmNvcmUuZmlsZXIuSGFzaEZpbGVyOwppbXBvcnQgb3JnLmFwYWNoZS54aW5kaWNlLmNv
cmUuaW5kZXhlci5JbmRleGVyOwppbXBvcnQgb3JnLmFwYWNoZS54aW5kaWNlLnV0aWwuQ29uZmln
dXJhdGlvbjsKaW1wb3J0IG9yZy5hcGFjaGUueGluZGljZS51dGlsLlhpbmRpY2VFeGNlcHRpb247
CmltcG9ydCBvcmcuYXBhY2hlLnhpbmRpY2UudXRpbC5SZWFkT25seUV4Y2VwdGlvbjsKaW1wb3J0
IG9yZy5hcGFjaGUueGluZGljZS54bWwuZG9tLkRPTVBhcnNlcjsKaW1wb3J0IG9yZy5hcGFjaGUu
eGluZGljZS5zZXJ2ZXIuWGluZGljZTsKCmltcG9ydCBqYXZhLmlvLkZpbGU7CmltcG9ydCBqYXZh
LmlvLklPRXhjZXB0aW9uOwoKLyoqCiAqIENvbW1hbmQgbGluZSB1dGlsaXR5IHRvIHJlLWJ1aWxk
IGFsbCBidHJlZSBhbmQgaGFzaCBmaWxlciBiYXNlZCBjb2xsZWN0aW9ucwogKiBvZiB0aGUgZGF0
YWJhc2UsIGFuZCByZS1pbmRleC4KICoKICogQHZlcnNpb24gJFJldmlzaW9uOiA1MzMyNTMgJCwg
JERhdGU6IDIwMDctMDQtMjcgMTk6MTI6MTIgLTA0MDAgKEZyaSwgMjcgQXByIDIwMDcpICQKICov
CnB1YmxpYyBjbGFzcyBEYXRhYmFzZVJlYnVpbGQgewoKICAgIHByaXZhdGUgc3RhdGljIGZpbmFs
IExvZyBsb2cgPSBMb2dGYWN0b3J5LmdldExvZyhEYXRhYmFzZVJlYnVpbGQuY2xhc3MpOwoKICAg
IHByaXZhdGUgc3RhdGljIGZpbmFsIGludCBDTURfQUxMICAgPSAwOwogICAgcHJpdmF0ZSBzdGF0
aWMgZmluYWwgaW50IENNRF9DT1BZICA9IDE7CiAgICBwcml2YXRlIHN0YXRpYyBmaW5hbCBpbnQg
Q01EX0lOREVYID0gMjsKCiAgICBwcml2YXRlIHN0YXRpYyBmaW5hbCBjaGFyIE5PVF9SQU4gPSAn
Xyc7CiAgICBwcml2YXRlIHN0YXRpYyBmaW5hbCBjaGFyIFNVQ0NFU1MgPSAnKic7CiAgICBwcml2
YXRlIHN0YXRpYyBmaW5hbCBjaGFyIEVSUk9SICAgPSAnISc7CgogICAgcHJpdmF0ZSBzdGF0aWMg
aW50IHBhZ2VDb3VudCA9IC0xOwoKICAgIHB1YmxpYyBzdGF0aWMgdm9pZCBtYWluKFN0cmluZ1td
IGFyZ3MpIHRocm93cyBFeGNlcHRpb24gewogICAgICAgIGlmIChhcmdzLmxlbmd0aCA8IDIgfHwg
YXJnc1sxXSA9PSBudWxsIHx8IGFyZ3NbMV0ubGVuZ3RoKCkgPT0gMCB8fAogICAgICAgICAgICAh
KCJjb3B5Ii5lcXVhbHMoYXJnc1swXSkgfHwgImluZGV4Ii5lcXVhbHMoYXJnc1swXSkgfHwgInJl
YnVpbGQiLmVxdWFscyhhcmdzWzBdKSkpIHsKICAgICAgICAgICAgdXNhZ2UoKTsKICAgICAgICAg
ICAgU3lzdGVtLmV4aXQoMSk7CiAgICAgICAgfQoKICAgICAgICBpbnQgY29tbWFuZDsKICAgICAg
ICBpZiAoImNvcHkiLmVxdWFscyhhcmdzWzBdKSkgewogICAgICAgICAgICBjb21tYW5kID0gQ01E
X0NPUFk7CiAgICAgICAgfSBlbHNlIGlmICgiaW5kZXgiLmVxdWFscyhhcmdzWzBdKSkgewogICAg
ICAgICAgICBjb21tYW5kID0gQ01EX0lOREVYOwogICAgICAgIH0gZWxzZSB7CiAgICAgICAgICAg
IGNvbW1hbmQgPSBDTURfQUxMOwogICAgICAgIH0KCiAgICAgICAgaWYgKGFyZ3MubGVuZ3RoID4g
MikgewogICAgICAgICAgICBpZiAoISgiLXAiLmVxdWFscyhhcmdzWzJdKSkgfHwgYXJncy5sZW5n
dGggPCA0KSB7CiAgICAgICAgICAgICAgICB1c2FnZSgpOwogICAgICAgICAgICAgICAgU3lzdGVt
LmV4aXQoMSk7CiAgICAgICAgICAgIH0KICAgICAgICAgICAgdHJ5IHsKICAgICAgICAgICAgICAg
IHBhZ2VDb3VudCA9IEludGVnZXIucGFyc2VJbnQoYXJnc1szXSk7CiAgICAgICAgICAgIH0gY2F0
Y2ggKE51bWJlckZvcm1hdEV4Y2VwdGlvbiBlKSB7CiAgICAgICAgICAgICAgICB1c2FnZSgpOwog
ICAgICAgICAgICAgICAgU3lzdGVtLmV4aXQoMSk7CiAgICAgICAgICAgIH0KICAgICAgICB9Cgog
ICAgICAgIEZpbGUgbG9jYXRpb24gPSBuZXcgRmlsZShhcmdzWzFdKTsKICAgICAgICBTdHJpbmcg
cGF0aCA9IGxvY2F0aW9uLmdldEFic29sdXRlUGF0aCgpOwogICAgICAgIFN0cmluZyBuYW1lID0g
bG9jYXRpb24uZ2V0TmFtZSgpOwoKICAgICAgICBpZiAoIiIuZXF1YWxzKG5hbWUpIHx8ICFsb2Nh
dGlvbi5leGlzdHMoKSB8fCAhbG9jYXRpb24uaXNEaXJlY3RvcnkoKSkgewogICAgICAgICAgICBT
eXN0ZW0ub3V0LnByaW50bG4oIkRhdGFiYXNlIHBhdGggbXVzdCBwb2ludCB0byBleGlzdGluZyBk
YXRhYmFzZSBkaXJlY3RvcnkiKTsKICAgICAgICAgICAgU3lzdGVtLmV4aXQoMSk7CiAgICAgICAg
fQoKICAgICAgICAvLyBjcmVhdGUgbWluaW1hbCBkYXRhYmFzZSBjb25maWd1cmF0aW9uIGluc3Rl
YWQgb2YgdHJ5aW5nIHRvIGxvY2F0ZSBzeXN0ZW0ueG1sCiAgICAgICAgU3RyaW5nIGNvbmZpZyA9
ICI8cm9vdC1jb2xsZWN0aW9uIGRicm9vdD0nIiArIHBhdGggKyAiLycgbmFtZT0nIiArIG5hbWUg
KyAiJy8+IjsKICAgICAgICBEYXRhYmFzZSBkYiA9IG5ldyBEYXRhYmFzZSgpOwoKICAgICAgICBi
b29sZWFuIHN1Y2Nlc3MgPSB0cnVlOwogICAgICAgIHRyeSB7CiAgICAgICAgICAgIFN5c3RlbS5v
dXQucHJpbnRsbigiUHJvY2Vzc2luZyBkYXRhYmFzZSAiICsgcGF0aCk7CiAgICAgICAgICAgIFN5
c3RlbS5vdXQucHJpbnRsbigpOwogICAgICAgICAgICBTeXN0ZW0ub3V0LnByaW50bG4oIkRJXHRD
b2xsZWN0aW9uIG5hbWUiKTsKCiAgICAgICAgICAgIGRiLnNldENvbmZpZyhuZXcgQ29uZmlndXJh
dGlvbihET01QYXJzZXIudG9Eb2N1bWVudChjb25maWcpKSk7CgogICAgICAgICAgICBpZiAobG9n
LmlzSW5mb0VuYWJsZWQoKSkgewogICAgICAgICAgICAgICAgbG9nLmluZm8oIlJlYnVpbGRpbmcg
Y29sbGVjdGlvbnMuLi4iKTsKICAgICAgICAgICAgfQogICAgICAgICAgICBzdWNjZXNzID0gcHJv
Y2Vzc0NvbGxlY3Rpb24oZGIsIGNvbW1hbmQpOwogICAgICAgIH0gZmluYWxseSB7CiAgICAgICAg
ICAgIGRiLmNsb3NlKCk7CgogICAgICAgICAgICBTeXN0ZW0ub3V0LnByaW50bG4oKTsKICAgICAg
ICAgICAgU3lzdGVtLm91dC5wcmludGxuKCJMZWdlbmQ6Iik7CiAgICAgICAgICAgIFN5c3RlbS5v
dXQucHJpbnRsbigiICAgIEQgRGF0YSBmaWxlIik7CiAgICAgICAgICAgIFN5c3RlbS5vdXQucHJp
bnRsbigiICAgIEkgSW5kZXggZmlsZXMiKTsKICAgICAgICAgICAgU3lzdGVtLm91dC5wcmludGxu
KCIgICAgXyBGaWxlIHdhcyBza2lwcGVkIik7CiAgICAgICAgICAgIFN5c3RlbS5vdXQucHJpbnRs
bigiICAgICogRmlsZSB3YXMgcHJvY2Vzc2VkIHN1Y2Nlc3NmdWx5Iik7CiAgICAgICAgICAgIFN5
c3RlbS5vdXQucHJpbnRsbigiICAgICEgRmlsZSBwcm9jZXNzaW5nIGZhaWxlZCIpOwoKICAgICAg
ICAgICAgU3lzdGVtLm91dC5wcmludGxuKCk7CiAgICAgICAgICAgIGlmIChzdWNjZXNzKSB7CiAg
ICAgICAgICAgICAgICBTeXN0ZW0ub3V0LnByaW50bG4oIlJlYnVpbGRpbmcgZGF0YWJhc2Ugd2Fz
IHN1Y2Nlc3NmdWxsLiIpOwogICAgICAgICAgICB9IGVsc2UgewogICAgICAgICAgICAgICAgU3lz
dGVtLm91dC5wcmludGxuKCJSZWJ1aWxkaW5nIGRhdGFiYXNlIGZhaWxlZC4gUGxlYXNlIGNoZWNr
IGxvZ3MgZm9yIG1vcmUgZGV0YWlsLiIpOwogICAgICAgICAgICB9CgogICAgICAgICAgICBTeXN0
ZW0uZXhpdChzdWNjZXNzID8gMCA6IDIpOwogICAgICAgIH0KCiAgICB9CgogICAgcHJpdmF0ZSBz
dGF0aWMgdm9pZCB1c2FnZSgpIHsKICAgICAgICBTeXN0ZW0ub3V0LnByaW50bG4oIlhpbmRpY2Ug
IiArIFhpbmRpY2UuVmVyc2lvbiArICIgRGF0YWJhc2UgUmVidWlsZCBVdGlsaXR5Iik7CiAgICAg
ICAgU3lzdGVtLm91dC5wcmludGxuKCJVc2FnZToiKTsKICAgICAgICBTeXN0ZW0ub3V0LnByaW50
bG4oIiAgICB4aW5kaWNlX3JlYnVpbGQgY29weSAgICA8ZGIgbG9jYXRpb24+IFstcCA8cGFnZWNv
dW50Pl0iKTsKICAgICAgICBTeXN0ZW0ub3V0LnByaW50bG4oIiAgICB4aW5kaWNlX3JlYnVpbGQg
aW5kZXggICA8ZGIgbG9jYXRpb24+Iik7CiAgICAgICAgU3lzdGVtLm91dC5wcmludGxuKCIgICAg
eGluZGljZV9yZWJ1aWxkIHJlYnVpbGQgPGRiIGxvY2F0aW9uPiBbLXAgPHBhZ2Vjb3VudD5dIik7
CiAgICAgICAgU3lzdGVtLm91dC5wcmludGxuKCk7CiAgICAgICAgU3lzdGVtLm91dC5wcmludGxu
KCJEQiBMb2NhdGlvbiBzaG91bGQgcG9pbnQgdG8gdGhlIGRpcmVjdG9yeSBjb250YWluaW5nIFhp
bmRpY2UgZGF0YWJhc2UgZmlsZXMuIik7CiAgICAgICAgU3lzdGVtLm91dC5wcmludGxuKCk7CiAg
ICAgICAgU3lzdGVtLm91dC5wcmludGxuKCJJbXBvcnRhbnQ6IFNodXRkb3duIGFuZCBiYWNrdXAg
ZGF0YWJhc2UgYmVmb3JlIHByb2NlZWRpbmchIik7CiAgICAgICAgU3lzdGVtLm91dC5wcmludGxu
KCk7CiAgICB9CgogICAgcHJpdmF0ZSBzdGF0aWMgYm9vbGVhbiBwcm9jZXNzQ29sbGVjdGlvbihD
b2xsZWN0aW9uIGNvbCwgaW50IGNvbW1hbmQpIHsKICAgICAgICBTdHJpbmcgbmFtZSA9IGNvbC5n
ZXRDYW5vbmljYWxOYW1lKCk7CiAgICAgICAgYm9vbGVhbiBzdGF0dXM7CgogICAgICAgIHRyeSB7
CiAgICAgICAgICAgIGlmIChsb2cuaXNJbmZvRW5hYmxlZCgpKSB7CiAgICAgICAgICAgICAgICBs
b2cuaW5mbygiUHJvY2Vzc2luZyBjb2xsZWN0aW9uICIgKyBuYW1lKTsKICAgICAgICAgICAgfQoK
ICAgICAgICAgICAgY2hhciBjb3B5ID0gTk9UX1JBTjsKICAgICAgICAgICAgY2hhciBpbmRleCA9
IE5PVF9SQU47CiAgICAgICAgICAgIHN3aXRjaCAoY29tbWFuZCkgewogICAgICAgICAgICAgICAg
Y2FzZSBDTURfQ09QWToKICAgICAgICAgICAgICAgICAgICBzdGF0dXMgPSByZWJ1aWxkQ29sbGVj
dGlvbihjb2wpOwogICAgICAgICAgICAgICAgICAgIGNvcHkgPSBzdGF0dXMgPyBTVUNDRVNTIDog
RVJST1I7CiAgICAgICAgICAgICAgICAgICAgYnJlYWs7CgogICAgICAgICAgICAgICAgY2FzZSBD
TURfSU5ERVg6CiAgICAgICAgICAgICAgICAgICAgc3RhdHVzID0gcmVidWlsZEluZGV4KGNvbCk7
CiAgICAgICAgICAgICAgICAgICAgaW5kZXggPSBzdGF0dXMgPyBTVUNDRVNTIDogRVJST1I7CiAg
ICAgICAgICAgICAgICAgICAgYnJlYWs7CiAgICAgICAgICAgICAgICBkZWZhdWx0OgogICAgICAg
ICAgICAgICAgICAgIHN0YXR1cyA9IHJlYnVpbGRDb2xsZWN0aW9uKGNvbCk7CiAgICAgICAgICAg
ICAgICAgICAgY29weSA9IHN0YXR1cyA/IFNVQ0NFU1MgOiBFUlJPUjsKICAgICAgICAgICAgICAg
ICAgICBpZiAoc3RhdHVzKSB7CiAgICAgICAgICAgICAgICAgICAgICAgIHN0YXR1cyA9IHJlYnVp
bGRJbmRleChjb2wpOwogICAgICAgICAgICAgICAgICAgICAgICBpbmRleCA9IHN0YXR1cyA/IFNV
Q0NFU1MgOiBFUlJPUjsKICAgICAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgICAgICAgICAg
YnJlYWs7CiAgICAgICAgICAgIH0KCiAgICAgICAgICAgIFN5c3RlbS5vdXQucHJpbnRsbihTdHJp
bmcudmFsdWVPZihjb3B5KSArIFN0cmluZy52YWx1ZU9mKGluZGV4KSArICJcdCIgKyBuYW1lKTsK
CiAgICAgICAgICAgIFN0cmluZ1tdIGNvbE5hbWVzID0gY29sLmxpc3RDb2xsZWN0aW9ucygpOwog
ICAgICAgICAgICBmb3IgKGludCBpID0gMDsgaSA8IGNvbE5hbWVzLmxlbmd0aDsgaSsrKSB7CiAg
ICAgICAgICAgICAgICBib29sZWFuIHJlc3VsdCA9IHByb2Nlc3NDb2xsZWN0aW9uKGNvbC5nZXRD
b2xsZWN0aW9uKGNvbE5hbWVzW2ldKSwgY29tbWFuZCk7CiAgICAgICAgICAgICAgICBzdGF0dXMg
PSBzdGF0dXMgJiYgcmVzdWx0OwogICAgICAgICAgICB9CiAgICAgICAgfSBjYXRjaCAoREJFeGNl
cHRpb24gZSkgewogICAgICAgICAgICBsb2cuZXJyb3IoIkdvdCBhbiBleGNlcHRpb24gd2hlbiBw
cm9jZXNzaW5nIGNvbGxlY3Rpb24gIiArIG5hbWUsIGUpOwoKICAgICAgICAgICAgcmV0dXJuIGZh
bHNlOwogICAgICAgIH0KCiAgICAgICAgcmV0dXJuIHN0YXR1czsKICAgIH0KCiAgICBwcml2YXRl
IHN0YXRpYyBib29sZWFuIHJlYnVpbGRDb2xsZWN0aW9uKENvbGxlY3Rpb24gY29sKSB7CiAgICAg
ICAgU3RyaW5nIGNhbm9uaWNhbE5hbWUgPSBjb2wuZ2V0Q2Fub25pY2FsTmFtZSgpOwoKICAgICAg
ICAvLyBjbG9zZSBjb2xsZWN0aW9uJ3MgZmlsZXIKICAgICAgICB0cnkgewogICAgICAgICAgICBp
ZiAoY29sLmdldEZpbGVyKCkgIT0gbnVsbCkgewogICAgICAgICAgICAgICAgY29sLmdldEZpbGVy
KCkuY2xvc2UoKTsKICAgICAgICAgICAgfQogICAgICAgIH0gY2F0Y2ggKERCRXhjZXB0aW9uIGUp
IHsKICAgICAgICAgICAgbG9nLmVycm9yKCJDb3VsZCBub3QgY2xvc2UgZmlsZXIgZm9yIGNvbGxl
Y3Rpb24gIiArIGNhbm9uaWNhbE5hbWUsIGUpOwogICAgICAgICAgICByZXR1cm4gZmFsc2U7CiAg
ICAgICAgfQoKICAgICAgICAvLyBwcmVwYXJlCiAgICAgICAgRmlsZXIgaXRzRmlsZXIgPSBjb2wu
Z2V0RmlsZXIoKTsKICAgICAgICBGaWxlckNvcHkgb2xkRmlsZXI7CiAgICAgICAgRmlsZXJDb3B5
IG5ld0ZpbGVyOwogICAgICAgIGlmIChpdHNGaWxlciA9PSBudWxsKSB7CiAgICAgICAgICAgIGlm
IChsb2cuaXNJbmZvRW5hYmxlZCgpKSB7CiAgICAgICAgICAgICAgICBsb2cuaW5mbygiQ29sbGVj
dGlvbiAiICsgY29sLmdldENhbm9uaWNhbE5hbWUoKSArICIgaGFzIG5vIGZpbGVyLiBTa2lwcGlu
Zy4uLiIpOwogICAgICAgICAgICB9CgogICAgICAgICAgICByZXR1cm4gdHJ1ZTsKICAgICAgICB9
IGlmIChpdHNGaWxlciBpbnN0YW5jZW9mIEJUcmVlRmlsZXIpIHsKICAgICAgICAgICAgb2xkRmls
ZXIgPSBuZXcgQlRyZWVDb3B5KCk7CiAgICAgICAgICAgIG5ld0ZpbGVyID0gbmV3IEJUcmVlQ29w
eSgpOwogICAgICAgIH0gZWxzZSBpZiAoaXRzRmlsZXIgaW5zdGFuY2VvZiBIYXNoRmlsZXIpIHsK
ICAgICAgICAgICAgb2xkRmlsZXIgPSBuZXcgSGFzaENvcHkoKTsKICAgICAgICAgICAgbmV3Rmls
ZXIgPSBuZXcgSGFzaENvcHkoKTsKICAgICAgICB9IGVsc2UgewogICAgICAgICAgICBpZiAobG9n
LmlzSW5mb0VuYWJsZWQoKSkgewogICAgICAgICAgICAgICAgbG9nLmluZm8oIkNvbGxlY3Rpb24g
IiArIGNvbC5nZXRDYW5vbmljYWxOYW1lKCkgKyAiIGhhcyB1bnJlY29nbml6ZWQgZmlsZXIgJyIg
KyBpdHNGaWxlci5nZXRDbGFzcygpLmdldE5hbWUoKSArICInLiBTa2lwcGluZy4uLiIpOwogICAg
ICAgICAgICB9CgogICAgICAgICAgICByZXR1cm4gdHJ1ZTsKICAgICAgICB9CgogICAgICAgIFN0
cmluZyBvbGRGaWxlTmFtZTsKICAgICAgICBTdHJpbmcgbmV3RmlsZU5hbWU7CiAgICAgICAgdHJ5
IHsKICAgICAgICAgICAgb2xkRmlsZXIuc2V0TG9jYXRpb24oY29sLmdldENvbGxlY3Rpb25Sb290
KCksIGNvbC5nZXROYW1lKCkpOwogICAgICAgICAgICBvbGRGaWxlci5zZXRDb25maWcoY29sLmdl
dEZpbGVyKCkuZ2V0Q29uZmlnKCkpOwogICAgICAgICAgICBvbGRGaWxlTmFtZSA9IG9sZEZpbGVy
LmdldEZpbGVyRmlsZSgpLmdldEFic29sdXRlUGF0aCgpOwogICAgICAgICAgICBpZiAoIW9sZEZp
bGVyLmV4aXN0cygpKSB7CiAgICAgICAgICAgICAgICBsb2cuZXJyb3IoIkZpbGVyIGZvciAiICsg
b2xkRmlsZU5hbWUgKyAiIGRvZXMgbm90IGV4aXN0cyIpOwogICAgICAgICAgICAgICAgcmV0dXJu
IGZhbHNlOwogICAgICAgICAgICB9CgogICAgICAgICAgICBuZXdGaWxlci5zZXRMb2NhdGlvbihj
b2wuZ2V0Q29sbGVjdGlvblJvb3QoKSwgY29sLmdldE5hbWUoKSArICIucmVidWlsZCIpOwogICAg
ICAgICAgICBuZXdGaWxlci5zZXRDb25maWcoY29sLmdldEZpbGVyKCkuZ2V0Q29uZmlnKCksIHBh
Z2VDb3VudCk7CiAgICAgICAgICAgIG5ld0ZpbGVOYW1lID0gbmV3RmlsZXIuZ2V0RmlsZXJGaWxl
KCkuZ2V0QWJzb2x1dGVQYXRoKCk7CiAgICAgICAgICAgIGlmIChuZXdGaWxlci5leGlzdHMoKSkg
ewogICAgICAgICAgICAgICAgbG9nLmVycm9yKCJGaWxlciBmb3IgIiArIG5ld0ZpbGVOYW1lICsg
IiBhbHJlYWR5IGV4aXN0cyIpOwogICAgICAgICAgICAgICAgcmV0dXJuIGZhbHNlOwogICAgICAg
ICAgICB9CiAgICAgICAgfSBjYXRjaCAoWGluZGljZUV4Y2VwdGlvbiBlKSB7CiAgICAgICAgICAg
IGxvZy5lcnJvcigiR290IGFuIGV4Y2VwdGlvbiB3aGVuIHByZXBhcmluZyB0byByZWJ1aWxkICIg
KyBjYW5vbmljYWxOYW1lLCBlKTsKICAgICAgICAgICAgcmV0dXJuIGZhbHNlOwogICAgICAgIH0K
CiAgICAgICAgLy8gY29weQogICAgICAgIGlmICghY29weShvbGRGaWxlciwgbmV3RmlsZXIsIGNh
bm9uaWNhbE5hbWUpKSB7CiAgICAgICAgICAgIG5ld0ZpbGVyLmRlbGV0ZUZpbGUoKTsKICAgICAg
ICAgICAgcmV0dXJuIGZhbHNlOwogICAgICAgIH0KCiAgICAgICAgb2xkRmlsZXIuZGVsZXRlRmls
ZSgpOwogICAgICAgIGlmICghbmV3RmlsZXIuZ2V0RmlsZXJGaWxlKCkucmVuYW1lVG8ob2xkRmls
ZXIuZ2V0RmlsZXJGaWxlKCkpKSB7CiAgICAgICAgICAgIGxvZy5lcnJvcigiQ291bGQgbm90IHJl
bmFtZSBzdWNjZXNzZnVsbHkgcmVidWlsdCBmaWxlICIgKyBuZXdGaWxlTmFtZSArICIgdG8gIiAr
IG9sZEZpbGVOYW1lKTsKICAgICAgICAgICAgcmV0dXJuIGZhbHNlOwogICAgICAgIH0KCiAgICAg
ICAgdHJ5IHsKICAgICAgICAgICAgY29sLmdldEZpbGVyKCkub3BlbigpOwogICAgICAgIH0gY2F0
Y2ggKERCRXhjZXB0aW9uIGUpIHsKICAgICAgICAgICAgbG9nLmVycm9yKCJDb3VsZCBub3Qgb3Bl
biBuZXcgZmlsZSAiICsgb2xkRmlsZU5hbWUsIGUpOwogICAgICAgICAgICByZXR1cm4gZmFsc2U7
CiAgICAgICAgfQoKICAgICAgICByZXR1cm4gdHJ1ZTsKICAgIH0KCiAgICBwcml2YXRlIHN0YXRp
YyBib29sZWFuIGNvcHkoRmlsZXJDb3B5IG9sZEZpbGVyLCBGaWxlckNvcHkgbmV3RmlsZXIsIFN0
cmluZyBjYW5vbmljYWxOYW1lKSB7CiAgICAgICAgdHJ5IHsKICAgICAgICAgICAgbmV3RmlsZXIu
Y3JlYXRlKCk7CiAgICAgICAgICAgIG9sZEZpbGVyLm9wZW4oKTsKICAgICAgICAgICAgbmV3Rmls
ZXIub3BlbigpOwogICAgICAgICAgICBvbGRGaWxlci5jb3B5KG5ld0ZpbGVyKTsKICAgICAgICB9
IGNhdGNoIChFeGNlcHRpb24gZSkgewogICAgICAgICAgICBsb2cuZXJyb3IoIkVycm9yIGNvcHlp
bmcgY29sbGVjdGlvbiAiICsgY2Fub25pY2FsTmFtZSwgZSk7CiAgICAgICAgICAgIHJldHVybiBm
YWxzZTsKICAgICAgICB9IGZpbmFsbHkgewogICAgICAgICAgICB0cnkgewogICAgICAgICAgICAg
ICAgb2xkRmlsZXIuY2xvc2UoKTsKICAgICAgICAgICAgfSBjYXRjaCAoREJFeGNlcHRpb24gZSkg
ewogICAgICAgICAgICAgICAgaWYgKGxvZy5pc1dhcm5FbmFibGVkKCkpIGxvZy53YXJuKGUpOwog
ICAgICAgICAgICB9CiAgICAgICAgICAgIHRyeSB7CiAgICAgICAgICAgICAgICBuZXdGaWxlci5j
bG9zZSgpOwogICAgICAgICAgICB9IGNhdGNoIChEQkV4Y2VwdGlvbiBlKSB7CiAgICAgICAgICAg
ICAgICBpZiAobG9nLmlzV2FybkVuYWJsZWQoKSkgbG9nLndhcm4oZSk7CiAgICAgICAgICAgIH0K
ICAgICAgICB9CgogICAgICAgIHJldHVybiB0cnVlOwogICAgfQoKICAgIHByaXZhdGUgc3RhdGlj
IGJvb2xlYW4gcmVidWlsZEluZGV4KENvbGxlY3Rpb24gY29sKSB7CiAgICAgICAgaWYgKGNvbC5n
ZXRGaWxlcigpID09IG51bGwpIHsKICAgICAgICAgICAgcmV0dXJuIHRydWU7CiAgICAgICAgfQoK
ICAgICAgICB0cnkgewogICAgICAgICAgICBTdHJpbmdbXSBsaXN0ID0gY29sLmxpc3RJbmRleGVy
cygpOwogICAgICAgICAgICBmb3IgKGludCBpID0gMDsgaSA8IGxpc3QubGVuZ3RoOyBpKyspIHsK
ICAgICAgICAgICAgICAgIEluZGV4ZXIgaWR4ID0gY29sLmdldEluZGV4ZXIobGlzdFtpXSk7CiAg
ICAgICAgICAgICAgICBDb25maWd1cmF0aW9uIGlkeENvbmYgPSBpZHguZ2V0Q29uZmlnKCk7CiAg
ICAgICAgICAgICAgICBpZiAobG9nLmlzSW5mb0VuYWJsZWQoKSkgewogICAgICAgICAgICAgICAg
ICAgIGxvZy5pbmZvKCJSZWJ1aWxkaW5nIGluZGV4ICIgKyBsaXN0W2ldICsgIiBmb3IgY29sbGVj
dGlvbiAiICsgY29sLmdldENhbm9uaWNhbE5hbWUoKSk7CiAgICAgICAgICAgICAgICB9CiAgICAg
ICAgICAgICAgICBjb2wuZHJvcEluZGV4ZXIoaWR4KTsKICAgICAgICAgICAgICAgIGNvbC5jcmVh
dGVJbmRleGVyKGlkeENvbmYpOwogICAgICAgICAgICB9CiAgICAgICAgfSBjYXRjaCAoREJFeGNl
cHRpb24gZSkgewogICAgICAgICAgICBsb2cuZXJyb3IoIkNvdWxkIG5vdCByZWJ1aWxkIGluZGV4
IGZvciBjb2xsZWN0aW9uICIgKyBjb2wuZ2V0Q2Fub25pY2FsTmFtZSgpLCBlKTsKICAgICAgICAg
ICAgcmV0dXJuIGZhbHNlOwogICAgICAgIH0KCiAgICAgICAgcmV0dXJuIHRydWU7CiAgICB9Cgog
ICAgcHJpdmF0ZSBpbnRlcmZhY2UgRmlsZXJDb3B5IGV4dGVuZHMgRmlsZXIgewogICAgICAgIHB1
YmxpYyBWYWx1ZSBnZXRWYWx1ZShsb25nIHBvaW50ZXIpIHRocm93cyBJT0V4Y2VwdGlvbjsKCiAg
ICAgICAgcHVibGljIHZvaWQgY29weShGaWxlciBuZXdGaWxlcikgdGhyb3dzIElPRXhjZXB0aW9u
LCBEQkV4Y2VwdGlvbjsKCiAgICAgICAgcHVibGljIGJvb2xlYW4gZGVsZXRlRmlsZSgpOwoKICAg
ICAgICBwdWJsaWMgRmlsZSBnZXRGaWxlckZpbGUoKTsKCiAgICAgICAgcHVibGljIHZvaWQgc2V0
Q29uZmlnKENvbmZpZ3VyYXRpb24gY29uZmlnLCBpbnQgcGFnZUNvdW50KTsKICAgIH0KCiAgICBw
cml2YXRlIHN0YXRpYyBjbGFzcyBCVHJlZUNvcHkgZXh0ZW5kcyBCVHJlZUZpbGVyIGltcGxlbWVu
dHMgRmlsZXJDb3B5IHsKICAgICAgICBwdWJsaWMgVmFsdWUgZ2V0VmFsdWUobG9uZyBwb2ludGVy
KSB0aHJvd3MgSU9FeGNlcHRpb24gewogICAgICAgICAgICByZXR1cm4gc3VwZXIucmVhZFZhbHVl
KHBvaW50ZXIpOwogICAgICAgIH0KCiAgICAgICAgcHVibGljIHZvaWQgY29weShGaWxlciBuZXdG
aWxlcikgdGhyb3dzIElPRXhjZXB0aW9uLCBCVHJlZUV4Y2VwdGlvbiB7CiAgICAgICAgICAgIHF1
ZXJ5KG51bGwsIG5ldyBDb3B5Q2FsbGJhY2sodGhpcywgbmV3RmlsZXIpKTsKICAgICAgICB9Cgog
ICAgICAgIHB1YmxpYyBib29sZWFuIGRlbGV0ZUZpbGUoKSB7CiAgICAgICAgICAgIHJldHVybiBn
ZXRGaWxlKCkuZGVsZXRlKCk7CiAgICAgICAgfQoKICAgICAgICBwdWJsaWMgRmlsZSBnZXRGaWxl
ckZpbGUoKSB7CiAgICAgICAgICAgIHJldHVybiBnZXRGaWxlKCk7CiAgICAgICAgfQoKICAgICAg
ICBwdWJsaWMgdm9pZCBzZXRDb25maWcoQ29uZmlndXJhdGlvbiBjb25maWcsIGludCBwYWdlQ291
bnQpIHsKICAgICAgICAgICAgaWYgKHBhZ2VDb3VudCA+IDApIHsKICAgICAgICAgICAgICAgIHRy
eSB7CiAgICAgICAgICAgICAgICAgICAgY29uZmlnLnNldEF0dHJpYnV0ZSgicGFnZWNvdW50Iiwg
cGFnZUNvdW50KTsKICAgICAgICAgICAgICAgIH0gY2F0Y2ggKFJlYWRPbmx5RXhjZXB0aW9uIGUp
IHsKICAgICAgICAgICAgICAgICAgICBsb2cuZXJyb3IoIkZhaWxlZCB0byBzZXQgY29uZmlndXJh
dGlvbiBwYXJhbWV0ZXIgJ3BhZ2Vjb3VudCciKTsKICAgICAgICAgICAgICAgIH0KICAgICAgICAg
ICAgfQoKICAgICAgICAgICAgc3VwZXIuc2V0Q29uZmlnKGNvbmZpZyk7CiAgICAgICAgfQogICAg
fQoKICAgIHByaXZhdGUgc3RhdGljIGNsYXNzIEhhc2hDb3B5IGV4dGVuZHMgSGFzaEZpbGVyIGlt
cGxlbWVudHMgRmlsZXJDb3B5IHsKICAgICAgICBwdWJsaWMgVmFsdWUgZ2V0VmFsdWUobG9uZyBw
b2ludGVyKSB0aHJvd3MgSU9FeGNlcHRpb24gewogICAgICAgICAgICByZXR1cm4gc3VwZXIucmVh
ZFZhbHVlKHBvaW50ZXIpOwogICAgICAgIH0KCiAgICAgICAgcHVibGljIHZvaWQgY29weShGaWxl
ciBuZXdGaWxlcikgdGhyb3dzIElPRXhjZXB0aW9uLCBEQkV4Y2VwdGlvbiB7CiAgICAgICAgICAg
IGxvbmcgaGFzaFNpemUgPSBnZXRGaWxlSGVhZGVyKCkuZ2V0UGFnZUNvdW50KCk7CgogICAgICAg
ICAgICBmb3IgKGxvbmcgaSA9IDA7IGkgPCBoYXNoU2l6ZTsgaSsrKSB7CiAgICAgICAgICAgICAg
ICBQYWdlIHBhZ2UgPSBnZXRQYWdlKGkpOwoKICAgICAgICAgICAgICAgIHdoaWxlICh0cnVlKSB7
CiAgICAgICAgICAgICAgICAgICAgSGFzaFBhZ2VIZWFkZXIgcGggPSAoSGFzaFBhZ2VIZWFkZXIp
IHBhZ2UuZ2V0UGFnZUhlYWRlcigpOwoKICAgICAgICAgICAgICAgICAgICBpZiAocGguZ2V0U3Rh
dHVzKCkgPT0gUkVDT1JEKSB7CiAgICAgICAgICAgICAgICAgICAgICAgIFZhbHVlIHZhbHVlID0g
cmVhZFZhbHVlKHBhZ2UpOwogICAgICAgICAgICAgICAgICAgICAgICBuZXdGaWxlci53cml0ZVJl
Y29yZChwYWdlLmdldEtleSgpLCB2YWx1ZSk7CgogICAgICAgICAgICAgICAgICAgICAgICBsb25n
IG5leHQgPSBwaC5nZXROZXh0Q29sbGlzaW9uKCk7CiAgICAgICAgICAgICAgICAgICAgICAgIGlm
IChuZXh0ICE9IE5PX1BBR0UpIHsKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHBhZ2UgPSBn
ZXRQYWdlKHBoLmdldE5leHRDb2xsaXNpb24oKSk7CiAgICAgICAgICAgICAgICAgICAgICAgIH0g
ZWxzZSB7CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBicmVhazsKICAgICAgICAgICAgICAg
ICAgICAgICAgfQogICAgICAgICAgICAgICAgICAgIH0gZWxzZSB7CiAgICAgICAgICAgICAgICAg
ICAgICAgIGJyZWFrOwogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgIH0KICAg
ICAgICAgICAgfQogICAgICAgIH0KCiAgICAgICAgcHVibGljIGJvb2xlYW4gZGVsZXRlRmlsZSgp
IHsKICAgICAgICAgICAgcmV0dXJuIGdldEZpbGUoKS5kZWxldGUoKTsKICAgICAgICB9CgogICAg
ICAgIHB1YmxpYyBGaWxlIGdldEZpbGVyRmlsZSgpIHsKICAgICAgICAgICAgcmV0dXJuIGdldEZp
bGUoKTsKICAgICAgICB9CgogICAgICAgIHB1YmxpYyB2b2lkIHNldENvbmZpZyhDb25maWd1cmF0
aW9uIGNvbmZpZywgaW50IHBhZ2VDb3VudCkgewogICAgICAgICAgICBzdXBlci5zZXRDb25maWco
Y29uZmlnKTsKICAgICAgICB9CiAgICB9CgogICAgcHJpdmF0ZSBzdGF0aWMgY2xhc3MgQ29weUNh
bGxiYWNrIGltcGxlbWVudHMgQlRyZWVDYWxsYmFjayB7CiAgICAgICAgcHJpdmF0ZSBCVHJlZUNv
cHkgZmlsZXI7CiAgICAgICAgcHJpdmF0ZSBGaWxlciBuZXdGaWxlcjsKCiAgICAgICAgcHVibGlj
IENvcHlDYWxsYmFjayhCVHJlZUNvcHkgZmlsZXIsIEZpbGVyIG5ld0ZpbGVyKSB7CiAgICAgICAg
ICAgIHRoaXMuZmlsZXIgPSBmaWxlcjsKICAgICAgICAgICAgdGhpcy5uZXdGaWxlciA9IG5ld0Zp
bGVyOwogICAgICAgIH0KCiAgICAgICAgcHVibGljIGJvb2xlYW4gaW5kZXhJbmZvKFZhbHVlIHZh
bHVlLCBsb25nIHBvaW50ZXIpIHsKICAgICAgICAgICAgdHJ5IHsKICAgICAgICAgICAgICAgIFZh
bHVlIHYgPSBmaWxlci5nZXRWYWx1ZShwb2ludGVyKTsKICAgICAgICAgICAgICAgIG5ld0ZpbGVy
LndyaXRlUmVjb3JkKG5ldyBLZXkodmFsdWUpLCB2KTsKICAgICAgICAgICAgfSBjYXRjaCAoRXhj
ZXB0aW9uIGUpIHsKICAgICAgICAgICAgICAgIGxvZy5lcnJvcihlKTsKICAgICAgICAgICAgfQoK
ICAgICAgICAgICAgcmV0dXJuIHRydWU7CiAgICAgICAgfQogICAgfQp9Cg==
--0016e64ea81e276ba10467c192b9--