Re: Script to find a best fit v11n
DM Smith <[email protected]> Thu, 19 Jun 2025 17:12:43 -0400
| Newsgroups | gmane.comp.literature.sword.devel |
|---|---|
| Message-ID | <[email protected]> |
--===============6396588308327274970== Content-Type: multipart/alternative; boundary="Apple-Mail=_968DFC0C-CFC9-4EEA-A345-B742A71BD0C6" --Apple-Mail=_968DFC0C-CFC9-4EEA-A345-B742A71BD0C6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jun 19, 2025, at 3:24=E2=80=AFPM, Greg Hellings = <[email protected]> wrote: >=20 >=20 >=20 > On Thu, Jun 19, 2025 at 9:07=E2=80=AFAM DM Smith = <[email protected] <mailto:[email protected]>> wrote: >> Greg, >> There=E2=80=99s an extraneous %s in the output. >=20 > Ah, not surprising. That is the old, Python 2 way of formatting = variables into a string, similar to C style printf syntax with variable = arguments coming in a tuple after an overload of the modulus operator = (so it would look like `"this is a string: %s" % (a_string, )` ). The = modern preferred way is with an f-string, where you preface a string = with the character `f` and then reference variables in the string with = {variable_name} syntax (e.g. `f"this is a string: {a_string}"`). That %s = can be killed off, or replaced with an f-string equivalent. > =20 >>=20 >> If you put the enumeration after the line "There are 93 OT IDs and 5 = NT IDs in v11n which aren=E2=80=99t in your file.=E2=80=9D Then you = wouldn=E2=80=99t need the heading "The following IDs don=E2=80=99t = appear in your file:=E2=80=9D >=20 > Yeah, I had been putting the IDs out to stderr with the logging = utility previously. It was only yesterday when I was squashing the = remaining Python 3 compat issues that I realized I should just drop them = into a print statement. They are, thusly, kinda crazy. In fact, I pass = them through a `sort` call, so they won't be in either canonical or = document order - unless the document has its verses sorted = alphabetically by osisID attribute for some inexplicable reason. > =20 >> It=E2=80=99d also be nice to format it a few per line, indented = appropriately. >=20 > Perhaps broken up by book? Or by book/chapter So it's like > Verses missing from: > Gen > 1 - 1, 3, 5, 7 > 2 - 11, 22 > Exo > 27 - 1 >=20 > There is a long way to go to improve the output, especially of this = detail portion. It was, after all, only intended as debugging output for = me while I was writing it. > =20 >>=20 >> I=E2=80=99d be happy to iterate over any suggestions we agree on. >=20 > As I am not a user of it, nor an intended consumer of it, feel free to = improve it as needed! I quickly hacked it together and tossed it out = into the world at someone's off-handed request. I don't create modules, = though, so I have no vested interest in preserving its current operation = in any particular form. And, if this thread has shown anything, it's = that likely Peter has been the only user to date. So I doubt you'll = disturb anyone else with it. >=20 > If you need my support for anything, I'm happy to lend a hand. >=20 > Pulling in comments from your other email on this thread: >=20 > > I like that it's very simple to read. Having a summary is good. And = the other email which lists the exact ids extra/missing per testament is = very helpful. > > I think that enumerating the names of the extra/missing books and = extra/missing chapters would be good. No sense in enumerating the ids = within these. >=20 > That probably would be good. I didn't include detection for an entire = missing chapter or book, but it shouldn't be too terribly difficult to = enhance it with that. A simple brute force check of every detected = missing book or chapter to see if there are any matched verses can = reveal that pretty easily. >=20 > > I ran mine against an input that was a test case for osis2mod=E2=80=99= s infinite loop and it had 2 extra books and 13 extra chapters. This = wouldn=E2=80=99t be obvious in your results. >=20 > True, mine would just complain about hundreds or even thousands of = mismatches and silently swallow the list of what those are. I had a few = of those that I omitted from the sample output I captured. For instance, = there are large portions of the canon for the Catholic versifications = missing from the KJV file. It just lists of something absurd like "There = are 4,741 missing verses" or whatever it is. >=20 > > Is it an advantage or disadvantage to be compiled against SWORD lib = vs slurping header files? >=20 > Like most things, it's a trade-off. Working with the bindings requires = that the Sword bindings are installed on the host system. For someone = running on Windows, this is particularly non-trivial. For someone = running in macOS it's not too difficult to install from source (I don't = believe Homebrew builds them). For users of major Linux distributions, = it's downright trivial. On Fedora it's as simple as a single `dnf = install python3-sword` command for a long time now, and it looks like = the bindings are also available for Ubuntu starting in 25.04 with an = `apt install python3-sword` as well. Regarding building SWORD on a Mac, I use homebrew for extra packages. I = tried to run ./autogen.sh, but it failed on libtoolize, which homebrew = doesn=E2=80=99t have. Then I ran cmake, which failed because icu4c = required C++17 or better. Hacking that I got CMakeLists.txt, I got it to = work. I=E2=80=99ll see if I can use that to run your script. > Advantages of the binding method are that it doesn't rely on parsing a = C header file, nor on the file laying out the values in a certain way. = It also can be used offline easily, doesn't require parsing the output = of HTML in order to find all the applicable files, and is likely = slightly faster. Not that the speed probably matters for a single run of = this, but if you're bulk processing files the speed advantages can add = up. The way I wrote mine is that it could use the include/canon*.h files = from a prior local SVN clone. This is very fast. I=E2=80=99d be curious = to see how it differs in speed from yours. The default is to go against = the web, which is painfully slow. (Note, it doesn=E2=80=99t yet do the = standard disclaimer for the web.) Not big deal if it is a single run. = Peter mentioned that he does additional analysis of the files in = problematic areas that cannot be done by the script. Using the python bindings does have the advantages of not re-inventing = the wheel. I was impressed with chatGPT=E2=80=99s regular expressions to = slurp the arrays and how concise it was to read the files. There really = wasn=E2=80=99t any difficulty in parsing the files. Since the canon*.h = files are very static and not likely to affect the parse. I don=E2=80=99t = think this is that big a deal. >=20 > Disadvantages of the binding method are that it's requiring you to = revert back to a source build if you are using this to test a canon.h = file or if you want to use a canon file that isn't available in the = package manager of your Linux distribution. Building from source isn't = terribly onerous for most of us contributors but it might be more of a = problem for a module maintainer. Then again, how often do we add a new = versification to the code base? So, it=E2=80=99s not something we=E2=80=99d expect a module maker to = succeed at if not on Un*x. Maybe someone has a library release for the = MacOS or Windows that could be used? >=20 > So there are pros and cons between them. I was freshly off of getting = the bindings to compile when I wrote the first draft of av11n.py so I = naturally went that direction. I also try to avoid writing parsers when = I can leverage existing ones, as grammars can be notoriously complex to = get correct. So that dictated my choices as much as did anything else, = really! My computer science masters degree was in compiler writing! It=E2=80=99s = definitely not for the faint of heart! >=20 > Another possible enhancement might be a CLI flag to limit the testing = range to a particular book (or testament) at a time. I have heard people = talk about having modules split up to one book per file or similar. If = they could say, "Only check this file against Joshua" then it could keep = down a significant amount of extra output. But again - I'm not really an = intended user of it! Great idea. So David=E2=80=99s suggestion of a scope argument. And I=E2=80=99m not an intended user of it either. I=E2=80=99m just = trying to get people to use something other than osis2mod to pick a = versification. Looking at the Jira issues on osis2mod, in one issue a = person listed their script that looped over the v11ns and called = osis2mod with each. Yuck! >=20 > --Greg >=20 >>=20 >> DM >>=20 >>> On Jun 19, 2025, at 12:12=E2=80=AFAM, Greg Hellings = <[email protected] <mailto:[email protected]>> wrote: >>>=20 >>> And here's an example now that I've fixed the output of the osisIDs = when there are fewer than 100 of them: >>>=20 >>> [vagrant@localhost ~]$ ./av11n.py kjv.osis.xml = =20 >>> = =20 >>> Checking Calvin: >>> ---------------- =20 >>> The following IDs don=E2=80=99t appear in your file: >>> %s 1Kgs.22.54, 1Sam.20.43, 1Sam.24.23, 3John.1.15, Acts.24.28, = Eccl.12.15, Eccl.12.16, Ezek.21.33, Ezek.21.34, Ezek.21.35, Ezek.21.36, = Ezek.21.37, Hos.12.15, Isa.8.23, Job.39.31, Job.39.32, Job.39.33, = Job.39.34, Job.39.35, Job.39.36, Job.39.37, Job.39.38 >>> , Job.40.25, Job.40.26, Job.40.27, Job.40.28, Jonah.2.11, = Mark.10.53, Mark.9.51, Num.13.34, Num.30.17, Ps.102.29, Ps.108.14, = Ps.12.9, Ps.140.14, Ps.142.8, Ps.18.51, Ps.19.15, Ps.20.10, Ps.21.14, = Ps.22.32, Ps.3.9, Ps.30.13, Ps.31.25, Ps.34.23, Ps.36.13, P >>> s.38.23, Ps.39.14, Ps.4.9, Ps.40.18, Ps.41.14, Ps.42.12, Ps.44.27, = Ps.45.18, Ps.46.12, Ps.47.10, Ps.48.15, Ps.49.21, Ps.5.13, Ps.51.20, = Ps.51.21, Ps.52.10, Ps.52.11, Ps.53.7, Ps.54.8, Ps.54.9, Ps.55.24, = Ps.56.14, Ps.57.12, Ps.58.12, Ps.59.18, Ps.6.11, Ps >>> .60.13, Ps.60.14, Ps.61.9, Ps.62.13, Ps.63.12, Ps.64.11, Ps.65.14, = Ps.67.8, Ps.68.36, Ps.69.37, Ps.7.18, Ps.70.6, Ps.75.11, Ps.76.13, = Ps.77.21, Ps.8.10, Ps.80.20, Ps.81.17, Ps.83.19, Ps.84.13, Ps.85.14, = Ps.88.19, Ps.89.53, Ps.9.21, Ps.92.16, Rev.12.18 >>> There are 93 OT IDs and 5 NT IDs in v11n which aren=E2=80=99t = in your file. >>> The following IDs don=E2=80=99t appear in v11n: = =20 >>> %s 1Kgs.22.54, 1Sam.20.43, 1Sam.24.23, 3John.1.15, Acts.24.28, = Eccl.12.15, Eccl.12.16, Ezek.21.33, Ezek.21.34, Ezek.21.35, Ezek.21.36, = Ezek.21.37, Hos.12.15, Isa.8.23, Job.39.31, Job.39.32, Job.39.33, = Job.39.34, Job.39.35, Job.39.36, Job.39.37, Job.39.38 >>> , Job.40.25, Job.40.26, Job.40.27, Job.40.28, Jonah.2.11, = Mark.10.53, Mark.9.51, Num.13.34, Num.30.17, Ps.102.29, Ps.108.14, = Ps.12.9, Ps.140.14, Ps.142.8, Ps.18.51, Ps.19.15, Ps.20.10, Ps.21.14, = Ps.22.32, Ps.3.9, Ps.30.13, Ps.31.25, Ps.34.23, Ps.36.13, P >>> s.38.23, Ps.39.14, Ps.4.9, Ps.40.18, Ps.41.14, Ps.42.12, Ps.44.27, = Ps.45.18, Ps.46.12, Ps.47.10, Ps.48.15, Ps.49.21, Ps.5.13, Ps.51.20, = Ps.51.21, Ps.52.10, Ps.52.11, Ps.53.7, Ps.54.8, Ps.54.9, Ps.55.24, = Ps.56.14, Ps.57.12, Ps.58.12, Ps.59.18, Ps.6.11, Ps >>> .60.13, Ps.60.14, Ps.61.9, Ps.62.13, Ps.63.12, Ps.64.11, Ps.65.14, = Ps.67.8, Ps.68.36, Ps.69.37, Ps.7.18, Ps.70.6, Ps.75.11, Ps.76.13, = Ps.77.21, Ps.8.10, Ps.80.20, Ps.81.17, Ps.83.19, Ps.84.13, Ps.85.14, = Ps.88.19, Ps.89.53, Ps.9.21, Ps.92.16, Rev.12.18 >>> There are 1 OT IDs and 29 NT IDs in your file which don=E2=80=99= t appear in v11n. >>>=20 >>>=20 >>> On Wed, Jun 18, 2025 at 11:00=E2=80=AFPM Greg Hellings = <[email protected] <mailto:[email protected]>> wrote: >>>> Here is an example of the first lines of running my script against = the kjv.osis.xml file from the git repo: >>>>=20 >>>>=20 >>>> Checking Calvin: >>>> ---------------- >>>> There are 93 OT IDs and 5 NT IDs in v11n which aren=E2=80=99t= in your file. >>>> There are 0 OT IDs and 30 NT IDs in your file which don=E2=80= =99t appear in v11n. >>>>=20 >>>> Checking Catholic: >>>> ------------------ >>>> There are 4530 OT IDs and 3 NT IDs in v11n which aren=E2=80=99= t in your file. >>>> There are 0 OT IDs and 133 NT IDs in your file which = don=E2=80=99t appear in v11n. >>>>=20 >>>> Checking Catholic2: >>>> ------------------- >>>> There are 4638 OT IDs and 3 NT IDs in v11n which aren=E2=80=99= t in your file. >>>> There are 0 OT IDs and 133 NT IDs in your file which = don=E2=80=99t appear in v11n. >>>>=20 >>>> Checking DarbyFr: >>>> ----------------- >>>> There are 31 OT IDs and 4 NT IDs in v11n which aren=E2=80=99t= in your file. >>>> There are 0 OT IDs and 30 NT IDs in your file which don=E2=80= =99t appear in v11n. >>>>=20 >>>> This continues on to include such output as >>>>=20 >>>> = =20 >>>> Checking KJV: >>>> -------------=20 >>>> Your file has all the references in this v11n >>>> Your file has no extra references = =20 >>>> = =20 >>>> Checking KJVA: =20 >>>> -------------- >>>> There are 5717 OT IDs and 0 NT IDs in v11n which aren=E2=80=99= t in your file. >>>> Your file has no extra references >>>>=20 >>>> giving a clear example of a winner for this particular file. >>>>=20 >>>> Meanwhile, running it against the kjva.osis.xml file includes this = in the results: >>>>=20 >>>> ... >>>>=20 >>>> Checking KJV: =20 >>>> ------------- =20 >>>> Your file has all the references in this v11n >>>> There are 2 OT IDs and 5715 NT IDs in your file which = don=E2=80=99t appear in v11n. >>>> =20 >>>> Checking KJVA: = =20 >>>> -------------- = =20 >>>> Your file has all the references in this v11n >>>> Your file has no extra references >>>> ... >>>>=20 >>>> Fiddling with the file has showed me there are a couple of places = where I need to tweak it for Python 3 compatibility that I missed the = last time I updated. But fixing those couple of little syntax issues = resulted in it running just fine in a Fedora 41 vm with nothing more to = do than invoke `dnf install python3-sword` to setup the system to use = it. >>>>=20 >>>> --Greg >>>>=20 >>>> On Wed, Jun 18, 2025 at 10:40=E2=80=AFPM Greg Hellings = <[email protected] <mailto:[email protected]>> wrote: >>>>> My script eschews percentages because they seemed relatively = pointless to me for measuring a mismatch like this. Instead it gives a = count of both Old and New Testament osisIDs that it finds missing and = another that it finds unexpectedly for a given versification. If the = total of either count is fewer than 100, the IDs for that particular = count are printed to the console. It will do this for every registered = versification in the version of the library it was compiled against, = allowing the user to select whichever one seems best to them based on = the results. >>>>>=20 >>>>> On Wed, Jun 18, 2025, 10:25=E2=80=AFPM David Haslam = <dfhdfh-g/[email protected] <mailto:dfhdfh-g/[email protected]>> wrote: >>>>>> It=E2=80=99s not just the number of =E2=80=9Cmissing=E2=80=9D = verses that should figure in the percentage score, but also the number = of verses that get concatenated to the last one in a chapter. >>>>>>=20 >>>>>> The differences in v11n for the Psalms will be especially = significant for this, in that some v11n renumber many of them. Likewise = for the last few chapters in the book of Job. >>>>>>=20 >>>>>> Aside: It would be cool to enhance the utility emptyvss by = providing a command line option that would ignore books that are not = included in the scope parameter in the conf file. >>>>>>=20 >>>>>> Regards, >>>>>>=20 >>>>>> David >>>>>>=20 >>>>>> On Thu, Jun 19, 2025 at 03:18, DM Smith <[email protected] = <mailto:On+Thu,+Jun+19,+2025+at+03:18,+DM+Smith+%3C%3Ca+href=3D>> wrote: >>>>>>>=20 >>>>>>> David, >>>>>>>=20 >>>>>>> Because it only considers the xml, scope is automatically built = into it. It is only comparing what is present in the xml with what is = part of the av11ns.=20 >>>>>>>=20 >>>>>>> It might be good to add the enumeration of missing verses. >>>>>>>=20 >>>>>>> =E2=80=94 DM >>>>>>>=20 >>>>>>>> On Jun 18, 2025, at 4:02=E2=80=AFPM, David Haslam = <dfhdfh-g/[email protected] <mailto:dfhdfh-g/[email protected]>> wrote: >>>>>>>>=20 >>>>>>>> Does it take account of the Scope key in the .conf file for a = less than complete Bible ? >>>>>>>>=20 >>>>>>>> David >>>>>>>>=20 >>>>>>>> Sent from Proton Mail <https://proton.me/mail/home> for iOS >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> On Wed, Jun 18, 2025 at 20:51, DM Smith < [email protected] = <mailto:On+Wed,+Jun+18,+2025+at+20:51,+DM+Smith+%3C%3Ca+href=3D>> wrote: >>>>>>>>>=20 >>>>>>>>> Hi, >>>>>>>>>=20 >>>>>>>>> Several have commented on how hard it is to test an OSIS xml = file against v11ns especially since it goes off into an infinite loop. = (I=E2=80=99ve posted a patch that fixes that) But it is still a process = of trial and error to find an appropriate v11n. >>>>>>>>>=20 >>>>>>>>> So, I=E2=80=99ve been iterating with chatGPT to create a = python script to find a best fit v11n. Since I don=E2=80=99t know = python, I can=E2=80=99t vouch for the script beyond it worked for a = simple test case that had an extra chapter for Genesis and had some = extra verses at the end of a chapter in that book. >>>>>>>>>=20 >>>>>>>>> I offer it, as a starting place. See the attached file. >>>>>>>>>=20 >>>>>>>>> It has a =E2=80=94debug flag. >>>>>>>>> The first argument is expected to be the OSIS xml file. >>>>>>>>> The second argument is optional and gives the location to the = include directory of svn/sword/trunk/include with all the canon*.h = files. If you don=E2=80=99t supply the argument, it uses the web to load = the canon*.h files from = https://www.crosswire.org/svn/sword/trunk/include.=20 >>>>>>>>>=20 >>>>>>>>> It will score the fitness of each of the v11ns. It gives the = score as a %, but I don=E2=80=99t know what that means. I told it that = it should prioritize book matches, then chapter matches and finally = verse matches. I don=E2=80=99t know how well it did that scoring. I = didn=E2=80=99t test for that. >>>>>>>>>=20 >>>>>>>>> The output is alphabetized. If more than one v11n have the = same high score, they are listed. >>>>>>>>>=20 >>>>>>>>> In His Service, >>>>>>>>> DM >>>>>>>>>=20 >>>>>>>> _______________________________________________=20 >>>>>>>> sword-devel mailing list: [email protected] = <mailto:[email protected]>=20 >>>>>>>> http://crosswire.org/mailman/listinfo/sword-devel=20 >>>>>>>> Instructions to unsubscribe/change your settings at above page >>>>>>>=20 >>>>>> _______________________________________________ >>>>>> sword-devel mailing list: [email protected] = <mailto:[email protected]> >>>>>> http://crosswire.org/mailman/listinfo/sword-devel >>>>>> Instructions to unsubscribe/change your settings at above page >>> _______________________________________________ >>> sword-devel mailing list: [email protected] = <mailto:[email protected]> >>> http://crosswire.org/mailman/listinfo/sword-devel >>> Instructions to unsubscribe/change your settings at above page >>=20 >> _______________________________________________ >> sword-devel mailing list: [email protected] = <mailto:[email protected]> >> http://crosswire.org/mailman/listinfo/sword-devel >> Instructions to unsubscribe/change your settings at above page > _______________________________________________ > sword-devel mailing list: [email protected] > http://crosswire.org/mailman/listinfo/sword-devel > Instructions to unsubscribe/change your settings at above page --Apple-Mail=_968DFC0C-CFC9-4EEA-A345-B742A71BD0C6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: = after-white-space;"><div><br></div><div><div><blockquote = type=3D"cite"><div>On Jun 19, 2025, at 3:24=E2=80=AFPM, Greg Hellings = <[email protected]> wrote:</div><br = class=3D"Apple-interchange-newline"><div><div dir=3D"ltr"><div = dir=3D"ltr"><br></div><br><div class=3D"gmail_quote = gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, Jun = 19, 2025 at 9:07=E2=80=AFAM DM Smith <<a = href=3D"mailto:[email protected]">[email protected]</a>> = wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div>Greg,<div>There=E2=80=99s an = extraneous %s in the = output.</div></div></blockquote><div><br></div><div>Ah, not surprising. = That is the old, Python 2 way of formatting variables into a string, = similar to C style printf syntax with variable arguments coming in a = tuple after an overload of the modulus operator (so it would look like = `"this is a string: %s" % (a_string, )` ). The modern preferred way is = with an f-string, where you preface a string with the character `f` and = then reference variables in the string with {variable_name} syntax (e.g. = `f"this is a string: {a_string}"`). That %s can be killed off, or = replaced with an f-string equivalent.</div><div> </div><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"><div><div><br></div><div>If you = put the enumeration after the line "There are 93 OT IDs and 5 NT IDs in = v11n which aren=E2=80=99t in your file.=E2=80=9D Then you wouldn=E2=80=99t= need the heading "The following IDs don=E2=80=99t appear in your = file:=E2=80=9D</div></div></blockquote><div><br></div><div>Yeah, I had = been putting the IDs out to stderr with the logging utility previously. = It was only yesterday when I was squashing the remaining Python 3 compat = issues that I realized I should just drop them into a print statement. = They are, thusly, kinda crazy. In fact, I pass them through a `sort` = call, so they won't be in either canonical or document order - unless = the document has its verses sorted alphabetically by osisID attribute = for some inexplicable reason.</div><div> </div><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"><div><div>It=E2=80=99d also be = nice to format it a few per line, indented = appropriately.</div></div></blockquote><div><br></div><div>Perhaps = broken up by book? Or by book/chapter So it's like</div><div>Verses = missing from:</div><div>Gen</div><div> 1 - 1, 3, 5, = 7</div><div> 2 - 11, 22</div><div>Exo</div><div> 27 - = 1</div><div><br></div><div>There is a long way to go to improve the = output, especially of this detail portion. It was, after all, only = intended as debugging output for me while I was writing = it.</div><div> </div><blockquote class=3D"gmail_quote" = style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div><div><br></div><div>I=E2=80=99d = be happy to iterate over any suggestions we agree = on.</div></div></blockquote><div><br></div><div>As I am not a user of = it, nor an intended consumer of it, feel free to improve it as needed! I = quickly hacked it together and tossed it out into the world at someone's = off-handed request. I don't create modules, though, so I have no vested = interest in preserving its current operation in any particular form. = And, if this thread has shown anything, it's that likely Peter has been = the only user to date. So I doubt you'll disturb anyone else with = it.</div><div><br></div><div>If you need my support for anything, I'm = happy to lend a hand.</div><div><br></div><div>Pulling in comments from = your other email on this thread:</div><div><br></div><div>> I like = that it's very simple to read. Having a summary is good. And the=20 other email which lists the exact ids extra/missing per testament is=20 very helpful.<div>> I think that enumerating the names of=20 the extra/missing books and extra/missing chapters would be good. No=20 sense in enumerating the ids within these.</div><div><br></div><div>That = probably would be good. I didn't include detection for an entire missing = chapter or book, but it shouldn't be too terribly difficult to enhance = it with that. A simple brute force check of every detected missing book = or chapter to see if there are any matched verses can reveal that pretty = easily.</div><div><br></div><div>> I=20 ran mine against an input that was a test case for osis2mod=E2=80=99s = infinite=20 loop and it had 2 extra books and 13 extra chapters. This wouldn=E2=80=99t= be=20 obvious in your results.</div><div><br></div><div>True, mine would just = complain about hundreds or even thousands of mismatches and silently = swallow the list of what those are. I had a few of those that I = omitted from the sample output I captured. For instance, there are large = portions of the canon for the Catholic versifications missing from the = KJV file. It just lists of something absurd like "There are = 4,741 missing verses" or whatever it is.</div><div><br></div><div>> = Is it an advantage or disadvantage to be compiled against SWORD lib vs = slurping header files?</div><div><br></div><div>Like most things, it's a = trade-off. Working with the bindings requires that the Sword bindings = are installed on the host system. For someone running on Windows, this = is particularly non-trivial. For someone running in macOS it's not too = difficult to install from source (I don't believe Homebrew builds them). = For users of major Linux distributions, it's downright trivial. On = Fedora it's as simple as a single `dnf install python3-sword` command = for a long time now, and it looks like the bindings are also available = for Ubuntu starting in 25.04 with an `apt install python3-sword` as = well.</div></div></div></div></div></blockquote><div><br></div>Regarding = building SWORD on a Mac, I use homebrew for extra packages. I tried to = run ./autogen.sh, but it failed on libtoolize, which homebrew doesn=E2=80=99= t have. Then I ran cmake, which failed because icu4c required C++17 or = better. Hacking that I got CMakeLists.txt, I got it to work. I=E2=80=99ll = see if I can use that to run your script.<div><br></div><blockquote = type=3D"cite"><div><div dir=3D"ltr"><div class=3D"gmail_quote = gmail_quote_container"><div><div>Advantages of the binding method are = that it doesn't rely on parsing a C header file, nor on the file laying = out the values in a certain way. It also can be used offline easily, = doesn't require parsing the output of HTML in order to find all the = applicable files, and is likely slightly faster. Not that the speed = probably matters for a single run of this, but if you're bulk processing = files the speed advantages can add = up.</div></div></div></div></div></blockquote><div><br></div>The way I = wrote mine is that it could use the include/canon*.h files from a prior = local SVN clone. This is very fast. I=E2=80=99d be curious to see how it = differs in speed from yours. The default is to go against the web, which = is painfully slow. (Note, it doesn=E2=80=99t yet do the standard = disclaimer for the web.) Not big deal if it is a single run. Peter = mentioned that he does additional analysis of the files in problematic = areas that cannot be done by the script.</div><div><br></div><div>Using = the python bindings does have the advantages of not re-inventing the = wheel. I was impressed with chatGPT=E2=80=99s regular expressions to = slurp the arrays and how concise it was to read the files. There really = wasn=E2=80=99t any difficulty in parsing the files. Since the canon*.h = files are very static and not likely to affect the parse. I don=E2=80=99t = think this is that big a deal.</div><div><br></div><div><br><blockquote = type=3D"cite"><div><div dir=3D"ltr"><div class=3D"gmail_quote = gmail_quote_container"><div><div><br></div><div>Disadvantages of the = binding method are that it's requiring you to revert back to a source = build if you are using this to test a canon.h file or if you want to use = a canon file that isn't available in the package manager of your Linux = distribution. Building from source isn't terribly onerous for most of us = contributors but it might be more of a problem for a module maintainer. = Then again, how often do we add a new versification to the code = base?</div></div></div></div></div></blockquote><div><br></div> So, = it=E2=80=99s not something we=E2=80=99d expect a module maker to succeed = at if not on Un*x. Maybe someone has a library release for the MacOS or = Windows that could be used?</div><div><br><blockquote = type=3D"cite"><div><div dir=3D"ltr"><div class=3D"gmail_quote = gmail_quote_container"><div><div><br></div><div>So there are pros and = cons between them. I was freshly off of getting the bindings to compile = when I wrote the first draft of av11n.py so I naturally went that = direction. I also try to avoid writing parsers when I can leverage = existing ones, as grammars can be notoriously complex to get correct. So = that dictated my choices as much as did anything else, = really!</div></div></div></div></div></blockquote><div><br></div>My = computer science masters degree was in compiler writing! It=E2=80=99s = definitely not for the faint of heart!</div><div><br><blockquote = type=3D"cite"><div><div dir=3D"ltr"><div class=3D"gmail_quote = gmail_quote_container"><div><div><br></div><div>Another possible = enhancement might be a CLI flag to limit the testing range to a = particular book (or testament) at a time. I have heard people talk about = having modules split up to one book per file or similar. If they could = say, "Only check this file against Joshua" then it could keep down a = significant amount of extra output. But again - I'm not really an = intended user of = it!</div></div></div></div></div></blockquote><div><br></div>Great idea. = So David=E2=80=99s suggestion of a scope = argument.</div><div><br></div><div>And I=E2=80=99m not an intended user = of it either. I=E2=80=99m just trying to get people to use something = other than osis2mod to pick a versification. Looking at the Jira issues = on osis2mod, in one issue a person listed their script that looped over = the v11ns and called osis2mod with each. Yuck!</div><div><br><blockquote = type=3D"cite"><div><div dir=3D"ltr"><div class=3D"gmail_quote = gmail_quote_container"><div><div><br></div><div>--Greg</div></div><div><br= ></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div><div><br></div><div>DM<br = id=3D"m_274868516123331404lineBreakAtBeginningOfMessage"><div><br><blockqu= ote type=3D"cite"><div>On Jun 19, 2025, at 12:12=E2=80=AFAM, Greg = Hellings <<a href=3D"mailto:[email protected]" = target=3D"_blank">[email protected]</a>> = wrote:</div><br><div><div dir=3D"ltr"><div>And here's an example now = that I've fixed the output of the osisIDs when there are fewer than 100 = of them:</div><div><br></div><div>[vagrant@localhost ~]$ ./av11n.py = kjv.osis.xml = = = = <br> = = = = = = <br>Checking = Calvin:<br>---------------- <br> The = following IDs don=E2=80=99t appear in your file:<br>%s 1Kgs.22.54, = 1Sam.20.43, 1Sam.24.23, 3John.1.15, Acts.24.28, Eccl.12.15, Eccl.12.16, = Ezek.21.33, Ezek.21.34, Ezek.21.35, Ezek.21.36, Ezek.21.37, Hos.12.15, = Isa.8.23, Job.39.31, Job.39.32, Job.39.33, Job.39.34, Job.39.35, = Job.39.36, Job.39.37, Job.39.38<br>, Job.40.25, Job.40.26, Job.40.27, = Job.40.28, Jonah.2.11, Mark.10.53, Mark.9.51, Num.13.34, Num.30.17, = Ps.102.29, Ps.108.14, Ps.12.9, Ps.140.14, Ps.142.8, Ps.18.51, Ps.19.15, = Ps.20.10, Ps.21.14, Ps.22.32, Ps.3.9, Ps.30.13, Ps.31.25, Ps.34.23, = Ps.36.13, P<br>s.38.23, Ps.39.14, Ps.4.9, Ps.40.18, Ps.41.14, Ps.42.12, = Ps.44.27, Ps.45.18, Ps.46.12, Ps.47.10, Ps.48.15, Ps.49.21, Ps.5.13, = Ps.51.20, Ps.51.21, Ps.52.10, Ps.52.11, Ps.53.7, Ps.54.8, Ps.54.9, = Ps.55.24, Ps.56.14, Ps.57.12, Ps.58.12, Ps.59.18, Ps.6.11, Ps<br>.60.13, = Ps.60.14, Ps.61.9, Ps.62.13, Ps.63.12, Ps.64.11, Ps.65.14, Ps.67.8, = Ps.68.36, Ps.69.37, Ps.7.18, Ps.70.6, Ps.75.11, Ps.76.13, Ps.77.21, = Ps.8.10, Ps.80.20, Ps.81.17, Ps.83.19, Ps.84.13, Ps.85.14, Ps.88.19, = Ps.89.53, Ps.9.21, Ps.92.16, Rev.12.18<br> = There are 93 OT IDs and 5 NT IDs in v11n which aren=E2=80=99t in your = file.<br> The following IDs don=E2=80=99t = appear in v11n: = = = = <br>%s 1Kgs.22.54, 1Sam.20.43, 1Sam.24.23, 3John.1.15, = Acts.24.28, Eccl.12.15, Eccl.12.16, Ezek.21.33, Ezek.21.34, Ezek.21.35, = Ezek.21.36, Ezek.21.37, Hos.12.15, Isa.8.23, Job.39.31, Job.39.32, = Job.39.33, Job.39.34, Job.39.35, Job.39.36, Job.39.37, Job.39.38<br>, = Job.40.25, Job.40.26, Job.40.27, Job.40.28, Jonah.2.11, Mark.10.53, = Mark.9.51, Num.13.34, Num.30.17, Ps.102.29, Ps.108.14, Ps.12.9, = Ps.140.14, Ps.142.8, Ps.18.51, Ps.19.15, Ps.20.10, Ps.21.14, Ps.22.32, = Ps.3.9, Ps.30.13, Ps.31.25, Ps.34.23, Ps.36.13, P<br>s.38.23, Ps.39.14, = Ps.4.9, Ps.40.18, Ps.41.14, Ps.42.12, Ps.44.27, Ps.45.18, Ps.46.12, = Ps.47.10, Ps.48.15, Ps.49.21, Ps.5.13, Ps.51.20, Ps.51.21, Ps.52.10, = Ps.52.11, Ps.53.7, Ps.54.8, Ps.54.9, Ps.55.24, Ps.56.14, Ps.57.12, = Ps.58.12, Ps.59.18, Ps.6.11, Ps<br>.60.13, Ps.60.14, Ps.61.9, Ps.62.13, = Ps.63.12, Ps.64.11, Ps.65.14, Ps.67.8, Ps.68.36, Ps.69.37, Ps.7.18, = Ps.70.6, Ps.75.11, Ps.76.13, Ps.77.21, Ps.8.10, Ps.80.20, Ps.81.17, = Ps.83.19, Ps.84.13, Ps.85.14, Ps.88.19, Ps.89.53, Ps.9.21, Ps.92.16, = Rev.12.18<br> There are 1 OT IDs and 29 NT = IDs in your file which don=E2=80=99t appear in = v11n.<br><br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Wed, Jun 18, 2025 at 11:00=E2=80=AFPM Greg = Hellings <<a href=3D"mailto:[email protected]" = target=3D"_blank">[email protected]</a>> = wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Here is an = example of the first lines of running my script against the kjv.osis.xml = file from the git repo:</div><div><br></div><div><br>Checking = Calvin:<br>----------------<br> There are 93 = OT IDs and 5 NT IDs in v11n which aren=E2=80=99t in your file.<br> = There are 0 OT IDs and 30 NT IDs in your file which = don=E2=80=99t appear in v11n.<br><br>Checking = Catholic:<br>------------------<br> There are = 4530 OT IDs and 3 NT IDs in v11n which aren=E2=80=99t in your = file.<br> There are 0 OT IDs and 133 NT IDs = in your file which don=E2=80=99t appear in v11n.<br><br>Checking = Catholic2:<br>-------------------<br> There = are 4638 OT IDs and 3 NT IDs in v11n which aren=E2=80=99t in your = file.<br> There are 0 OT IDs and 133 NT IDs = in your file which don=E2=80=99t appear in v11n.<br><br>Checking = DarbyFr:<br>-----------------<br> There are = 31 OT IDs and 4 NT IDs in v11n which aren=E2=80=99t in your = file.<br> There are 0 OT IDs and 30 NT IDs in = your file which don=E2=80=99t appear in = v11n.<br></div><div><br></div><div>This continues on to include such = output as</div><div><br></div><div> = = = = = = = <br>Checking KJV:<br>------------- <br> Your = file has all the references in this v11n<br> = Your file has no extra references = = = = <br> = = = = = = = <br>Checking KJVA: = <br>--------------<br> There are 5717 OT IDs = and 0 NT IDs in v11n which aren=E2=80=99t in your file.<br> = Your file has no extra references<br><br></div><div>giving = a clear example of a winner for this particular = file.</div><div><br></div><div>Meanwhile, running it against the = kjva.osis.xml file includes this in the = results:</div><div><br></div><div>...</div><div><br>Checking KJV: = <br>------------- = <br> Your file has all the references = in this v11n<br> There are 2 OT IDs and 5715 = NT IDs in your file which don=E2=80=99t appear in v11n.<br> = = = = <br>Checking KJVA: = = = = = = <br>-------------- = = = = = = <br> Your file has all the references in this = v11n<br> Your file has no extra = references</div><div>...</div><div><br></div><div>Fiddling with the file = has showed me there are a couple of places where I need to tweak it for = Python 3 compatibility that I missed the last time I updated. But fixing = those couple of little syntax issues resulted in it running just fine in = a Fedora 41 vm with nothing more to do than invoke `dnf install = python3-sword` to setup the system to use = it.</div><div><br></div><div>--Greg</div></div><br><div = class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Jun = 18, 2025 at 10:40=E2=80=AFPM Greg Hellings <<a = href=3D"mailto:[email protected]" = target=3D"_blank">[email protected]</a>> = wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div dir=3D"auto">My script eschews = percentages because they seemed relatively pointless to me for measuring = a mismatch like this. Instead it gives a count of both Old and New = Testament osisIDs that it finds missing and another that it finds = unexpectedly for a given versification. If the total of either count is = fewer than 100, the IDs for that particular count are printed to the = console. It will do this for every registered versification in the = version of the library it was compiled against, allowing the user to = select whichever one seems best to them based on the = results.</div><br><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Wed, Jun 18, 2025, 10:25=E2=80=AFPM David Haslam = <<a href=3D"mailto:dfhdfh-g/[email protected]" = target=3D"_blank">dfhdfh-g/[email protected]</a>> = wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div> <div dir=3D"auto">It=E2=80=99s = not just the number of =E2=80=9Cmissing=E2=80=9D verses that should = figure in the percentage score, but also the number of verses that get = concatenated to the last one in a chapter.</div><div = dir=3D"auto"><br></div><div dir=3D"auto">The differences in v11n for the = Psalms will be especially significant for this, in that some v11n = renumber many of them. Likewise for the last few chapters in the book of = Job.</div><div dir=3D"auto"><br></div><div dir=3D"auto">Aside: It would = be cool to enhance the utility emptyvss by providing a command line = option that would ignore books that are not included in the scope = parameter in the conf file.</div><div dir=3D"auto"><br></div><div = dir=3D"auto">Regards,</div><div><br></div> <div = dir=3D"auto">David</div><div><br></div>On Thu, Jun 19, 2025 at 03:18, DM = Smith <<a = href=3D"mailto:On+Thu,+Jun+19,+2025+at+03:18,+DM+Smith+%3C%3Ca+href=3D" = rel=3D"noreferrer" target=3D"_blank">[email protected]</a>> = wrote:<blockquote type=3D"cite"> <div> David, </div> <div> <br> </div> <div> Because it only considers the xml, scope is automatically built into = it. It is only comparing what is present in the xml with what is part of = the av11ns. </div> <div> <br> </div> <div> It might be good to add the enumeration of missing verses. </div> <div> <br> </div> <div> =E2=80=94 DM </div> <div> <br> <blockquote type=3D"cite"> <div> On Jun 18, 2025, at 4:02=E2=80=AFPM, David Haslam <<a = href=3D"mailto:dfhdfh-g/[email protected]" rel=3D"noreferrer" = target=3D"_blank">dfhdfh-g/[email protected]</a>> wrote: </div> <br> <div> <div>=20 <div dir=3D"auto"> Does it take account of the Scope key in the .conf file for a less = than complete Bible ? </div> <div dir=3D"auto"> <br> </div> <div dir=3D"auto"> David </div> <div> <br> </div>=20 <div = id=3D"m_274868516123331404m_-4072912897035575119m_-1962289800233906647m_-4= 553660002178259517protonmail_mobile_signature_block"> <div> Sent from=20 <a href=3D"https://proton.me/mail/home" rel=3D"noreferrer" = target=3D"_blank">Proton Mail</a> for iOS </div> </div>=20 <div> <br> </div> <div> <br> </div>On Wed, Jun 18, 2025 at 20:51, DM Smith < <a = href=3D"mailto:On+Wed,+Jun+18,+2025+at+20:51,+DM+Smith+%3C%3Ca+href=3D" = rel=3D"noreferrer" target=3D"_blank">[email protected]</a>> = wrote: <blockquote type=3D"cite"> Hi,=20 <div>=20 <br>=20 </div>=20 <div> Several have commented on how hard it is to test an OSIS xml file = against v11ns especially since it goes off into an infinite loop. = (I=E2=80=99ve posted a patch that fixes that) But it is still a process = of trial and error to find an appropriate v11n.=20 </div>=20 <div>=20 <br>=20 </div>=20 <div>=20 <div> So, I=E2=80=99ve been iterating with chatGPT to create a python = script to find a best fit v11n. Since I don=E2=80=99t know python, I = can=E2=80=99t vouch for the script beyond it worked for a simple test = case that had an extra chapter for Genesis and had some extra verses at = the end of a chapter in that book.=20 </div>=20 <div>=20 <br>=20 </div>=20 <div> I offer it, as a starting place. See the attached file.=20 </div>=20 <div>=20 <br>=20 </div>=20 <div> It has a =E2=80=94debug flag.=20 </div>=20 <div> The first argument is expected to be the OSIS xml file.=20 </div>=20 <div> The second argument is optional and gives the location to the = include directory of svn/sword/trunk/include with all the canon*.h = files. If you don=E2=80=99t supply the argument, it uses the web to load = the canon*.h files from=20 <a href=3D"https://www.crosswire.org/svn/sword/trunk/include" = rel=3D"noreferrer" = target=3D"_blank">https://www.crosswire.org/svn/sword/trunk/include</a>.&n= bsp;=20 </div>=20 <div>=20 <br>=20 </div>=20 <div> It will score the fitness of each of the v11ns. It gives the = score as a %, but I don=E2=80=99t know what that means. I told it that = it should prioritize book matches, then chapter matches and finally = verse matches. I don=E2=80=99t know how well it did that scoring. I = didn=E2=80=99t test for that.=20 </div>=20 <div>=20 <br>=20 </div>=20 <div> The output is alphabetized. If more than one v11n have the same = high score, they are listed.=20 </div>=20 <div>=20 <br>=20 </div>=20 <div> In His Service,=20 </div>=20 <div>=20 <span style=3D"white-space:pre-wrap"> </span>DM=20 </div>=20 <div>=20 <br>=20 </div>=20 <div></div>=20 </div>=20 <div>=20 <div></div>=20 </div> </blockquote> </div>_______________________________________________ <br>sword-devel mailing list: <a = href=3D"mailto:[email protected]" rel=3D"noreferrer" = target=3D"_blank">[email protected]</a> <br><a href=3D"http://crosswire.org/mailman/listinfo/sword-devel" = rel=3D"noreferrer" = target=3D"_blank">http://crosswire.org/mailman/listinfo/sword-devel</a> <br>Instructions to unsubscribe/change your settings at above page <br> </div> </blockquote> </div> = <br></blockquote></div>_______________________________________________<br>= sword-devel mailing list: <a href=3D"mailto:[email protected]" = rel=3D"noreferrer" target=3D"_blank">[email protected]</a><br> <a href=3D"http://crosswire.org/mailman/listinfo/sword-devel" = rel=3D"noreferrer noreferrer" = target=3D"_blank">http://crosswire.org/mailman/listinfo/sword-devel</a><br= > Instructions to unsubscribe/change your settings at above page<br> </blockquote></div> </blockquote></div> </blockquote></div> _______________________________________________<br>sword-devel mailing = list: <a href=3D"mailto:[email protected]" = target=3D"_blank">[email protected]</a><br><a = href=3D"http://crosswire.org/mailman/listinfo/sword-devel" = target=3D"_blank">http://crosswire.org/mailman/listinfo/sword-devel</a><br= >Instructions to unsubscribe/change your settings at above = page<br></div></blockquote></div><br></div></div>_________________________= ______________________<br> sword-devel mailing list: <a href=3D"mailto:[email protected]" = target=3D"_blank">[email protected]</a><br> <a href=3D"http://crosswire.org/mailman/listinfo/sword-devel" = rel=3D"noreferrer" = target=3D"_blank">http://crosswire.org/mailman/listinfo/sword-devel</a><br= > Instructions to unsubscribe/change your settings at above page<br> </blockquote></div></div> _______________________________________________<br>sword-devel mailing = list: = [email protected]<br>http://crosswire.org/mailman/listinfo/sword-d= evel<br>Instructions to unsubscribe/change your settings at above = page<br></div></blockquote></div><br></div></body></html>= --Apple-Mail=_968DFC0C-CFC9-4EEA-A345-B742A71BD0C6-- --===============6396588308327274970== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ sword-devel mailing list: [email protected] http://crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page --===============6396588308327274970==--