Re: Script to find a best fit v11n
Greg Hellings <[email protected]> Thu, 19 Jun 2025 22:20:49 -0500
| Newsgroups | gmane.comp.literature.sword.devel |
|---|---|
| Message-ID | <CAHxvOVKm2cZ8F8cipX9fYWTU83MZEaNnyO=YWYGg=SpbY9+jPg@mail.gmail.com> |
--===============6647260732657178196== Content-Type: multipart/alternative; boundary="000000000000f923b20637f85922" --000000000000f923b20637f85922 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Jun 19, 2025 at 4:13=E2=80=AFPM DM Smith <[email protected]> wr= ote: > > On Jun 19, 2025, at 3:24=E2=80=AFPM, Greg Hellings <[email protected]= om> > wrote: > > > Like most things, it's a trade-off. Working with the bindings requires > that the Sword bindings are installed on the host system. For someone > running on Windows, this is particularly non-trivial. For someone running > in macOS it's not too difficult to install from source (I don't believe > Homebrew builds them). For users of major Linux distributions, it's > downright trivial. On Fedora it's as simple as a single `dnf install > python3-sword` command for a long time now, and it looks like the binding= s > are also available for Ubuntu starting in 25.04 with an `apt install > python3-sword` as well. > > > Regarding building SWORD on a Mac, I use homebrew for extra packages. I > tried to run ./autogen.sh, but it failed on libtoolize, which homebrew > doesn=E2=80=99t have. Then I ran cmake, which failed because icu4c requir= ed C++17 > or better. Hacking that I got CMakeLists.txt, I got it to work. I=E2=80= =99ll see if > I can use that to run your script. > For these purposes, neither ICU nor CLucene are needed. It's only pulling the versification data which is core to the library's builtins. > > Advantages of the binding method are that it doesn't rely on parsing a C > header file, nor on the file laying out the values in a certain way. It > also can be used offline easily, doesn't require parsing the output of HT= ML > in order to find all the applicable files, and is likely slightly faster. > Not that the speed probably matters for a single run of this, but if you'= re > bulk processing files the speed advantages can add up. > > > The way I wrote mine is that it could use the include/canon*.h files from > a prior local SVN clone. This is very fast. I=E2=80=99d be curious to see= how it > differs in speed from yours. The default is to go against the web, which = is > painfully slow. (Note, it doesn=E2=80=99t yet do the standard disclaimer = for the > web.) Not big deal if it is a single run. Peter mentioned that he does > additional analysis of the files in problematic areas that cannot be done > by the script. > > Using the python bindings does have the advantages of not re-inventing th= e > wheel. I was impressed with chatGPT=E2=80=99s regular expressions to slur= p the > arrays and how concise it was to read the files. There really wasn=E2=80= =99t any > difficulty in parsing the files. Since the canon*.h files are very static > and not likely to affect the parse. I don=E2=80=99t think this is that bi= g a deal. > Yeah, the canon header files are pretty well structured following a standard format to make it easier on humans, and thus regex, to swallow. The thought of doing so had simply never crossed my mind. > > > > Disadvantages of the binding method are that it's requiring you to revert > back to a source build if you are using this to test a canon.h file or if > you want to use a canon file that isn't available in the package manager = of > your Linux distribution. Building from source isn't terribly onerous for > most of us contributors but it might be more of a problem for a module > maintainer. Then again, how often do we add a new versification to the co= de > base? > > > So, it=E2=80=99s not something we=E2=80=99d expect a module maker to suc= ceed at if not on > Un*x. Maybe someone has a library release for the MacOS or Windows that > could be used? > Because our Python bindings are built as part of the library and generated by Swig, they aren't distributed onto PyPI (the PYthon Package Index), which is the standard way of installing third party Python modules. To install from PyPI, one simply uses the "pip" tool, or other standard Python package installers. But for ours, the binding code is generated by Swig from the library code and we don't then subsequently distribute the module separately. Doing so would not be terribly difficult, but it is not a route we have taken previously. Of course, installing Python modules that include C bindings necessitates having the Python.h file available as well as a compatible version of a C compiler. For the official Python distributions, this is always and only MSVC - or at least it has been in the past. Officially Python has not historically even supported building for Windows with gcc. It's enough of a bugbear that I've never even bothered with installing modules with Python on Windows. Nowadays, though, we don't really need to. Anyone who wants to can install Ubuntu under the WSL and just take advantage of the existing apt package and Python in there. As for macOS, I haven't a good solution there. I only use it as demanded for work. Probably best to just let people who want it compile it from source there, and let them know there isn't any need for the ICU add ons. An alternative is to go beyond a Python script and create a full utility in C that does this same work. That would make distribution much easier to all of the platforms. The reason I did not initially take that route is that Python is so convenient for working with XML in whereas the library has no such mechanism to readily parse it and query in the same way. Obviously it can be done, as osis2mod is already doing that work. Its parsing code could be repurposed to this effect. > > So there are pros and cons between them. I was freshly off of getting the > bindings to compile when I wrote the first draft of av11n.py so I natural= ly > went that direction. I also try to avoid writing parsers when I can > leverage existing ones, as grammars can be notoriously complex to get > correct. So that dictated my choices as much as did anything else, really= ! > > > My computer science masters degree was in compiler writing! It=E2=80=99s > definitely not for the faint of heart! > Mine was in AI. Also not for the faint of heart, but much more approachable of a consumer product. > > > Another possible enhancement might be a CLI flag to limit the testing > range to a particular book (or testament) at a time. I have heard people > talk about having modules split up to one book per file or similar. If th= ey > could say, "Only check this file against Joshua" then it could keep down = a > significant amount of extra output. But again - I'm not really an intende= d > user of it! > > > Great idea. So David=E2=80=99s suggestion of a scope argument. > Basically, yes. > > And I=E2=80=99m not an intended user of it either. I=E2=80=99m just tryin= g to get people > to use something other than osis2mod to pick a versification. Looking at > the Jira issues on osis2mod, in one issue a person listed their script th= at > looped over the v11ns and called osis2mod with each. Yuck! > Yeah, using osis2mod in that way seems fraught with trouble. But in the absence of knowing more about the internal of the library, I can see why someone would take that approach. --Greg --000000000000f923b20637f85922 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g= mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, Jun 19,= 2025 at 4:13=E2=80=AFPM DM Smith <<a href=3D"mailto:[email protected]= rg">[email protected]</a>> wrote:<br></div><blockquote class=3D"gmai= l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20= 4,204);padding-left:1ex"><div><div><br></div><div><div><blockquote type=3D"= cite"><div>On Jun 19, 2025, at 3:24=E2=80=AFPM, Greg Hellings <<a href= =3D"mailto:[email protected]" target=3D"_blank">[email protected]= om</a>> wrote:</div><br><div><div dir=3D"ltr"><div dir=3D"ltr"><br></div= ><div class=3D"gmail_quote"><div><div>Like most things, it's a trade-of= f. Working with the bindings requires that the Sword bindings are installed= on the host system. For someone running on Windows, this is particularly n= on-trivial. For someone running in macOS it's not too difficult to inst= all from source (I don't believe Homebrew builds them). For users of ma= jor Linux distributions, it's downright trivial. On Fedora it's as = simple as a single `dnf install python3-sword` command for a long time now,= and it looks like the bindings are also available for Ubuntu starting in 2= 5.04 with an `apt install python3-sword` as well.</div></div></div></div></= div></blockquote><div><br></div>Regarding building SWORD on a Mac, I use ho= mebrew for extra packages. I tried to run ./autogen.sh, but it failed on li= btoolize, which homebrew doesn=E2=80=99t have. Then I ran cmake, which fail= ed because icu4c required C++17 or better. Hacking that I got CMakeLists.tx= t, I got it to work. I=E2=80=99ll see if I can use that to run your script.= </div></div></div></blockquote><div><br></div><div>For these purposes, neit= her ICU nor CLucene are needed. It's only pulling the versification dat= a which is core to the library's builtins.</div><div>=C2=A0</div><block= quote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1= px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div><br></div><= blockquote type=3D"cite"><div><div dir=3D"ltr"><div class=3D"gmail_quote"><= div><div>Advantages of the binding method are that it doesn't rely on p= arsing a C header file, nor on the file laying out the values in a certain = way. It also can be used offline easily, doesn't require parsing the ou= tput of HTML in order to find all the applicable files, and is likely sligh= tly faster. Not that the speed probably matters for a single run of this, b= ut if you're bulk processing files the speed advantages can add up.</di= v></div></div></div></div></blockquote><div><br></div>The way I wrote mine = is that it could use the include/canon*.h files from a prior local SVN clon= e. This is very fast. I=E2=80=99d be curious to see how it differs in speed= from yours. The default is to go against the web, which is painfully slow.= (Note, it doesn=E2=80=99t yet do the standard disclaimer for the web.) Not= big deal if it is a single run. Peter mentioned that he does additional an= alysis of the files in problematic areas that cannot be done by the script.= </div><div><br></div><div>Using the python bindings does have the advantage= s of not re-inventing the wheel. I was impressed with chatGPT=E2=80=99s reg= ular expressions to slurp the arrays and how concise it was to read the fil= es. There really wasn=E2=80=99t any difficulty in parsing the files. Since = the canon*.h files are very static and not likely to affect the parse. I do= n=E2=80=99t think this is that big a deal.</div></div></div></blockquote><d= iv><br></div><div>Yeah, the canon header files are pretty well structured f= ollowing a standard format to make it easier on humans, and thus regex, to = swallow. The thought of doing so had simply never crossed my mind.</div><di= v>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px= 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><= div><br></div><div><br><blockquote type=3D"cite"><div><div dir=3D"ltr"><div= class=3D"gmail_quote"><div><div><br></div><div>Disadvantages of the bindin= g method are that it's requiring you to revert back to a source build i= f you are using this to test a canon.h file or if you want to use a canon f= ile that isn't available in the package manager of your Linux distribut= ion. Building from source isn't terribly onerous for most of us contrib= utors but it might be more of a problem for a module maintainer. Then again= , how often do we add a new versification to the code base?</div></div></di= v></div></div></blockquote><div><br></div>=C2=A0So, it=E2=80=99s not someth= ing we=E2=80=99d expect a module maker to succeed at if not on Un*x. Maybe = someone has a library release for the MacOS or Windows that could be used?<= /div></div></div></blockquote><div><br></div><div>Because our Python bindin= gs are built as part of the library and generated by Swig, they aren't = distributed onto PyPI (the PYthon Package Index), which is the standard way= of installing third party Python modules. To install from PyPI, one simply= uses the "pip" tool, or other standard Python package installers= . But for ours, the binding code is generated by Swig from the library code= and we don't then subsequently distribute the module separately. Doing= so would not be terribly difficult, but it is not a route we have taken pr= eviously.</div><div><br></div><div>Of course, installing Python modules tha= t include C bindings necessitates having the Python.h file available as wel= l as a compatible version of a C compiler. For the official Python distribu= tions, this is always and only MSVC - or at least it has been in the past. = Officially Python has not historically even supported building for Windows = with gcc. It's enough of a bugbear that I've never even bothered wi= th installing modules with Python on Windows.</div><div><br></div><div>Nowa= days, though, we don't really need to. Anyone who wants to can install = Ubuntu under the WSL and just take advantage of the existing apt package an= d Python in there. As for macOS, I haven't a good solution there. I onl= y use it as demanded for work. Probably best to just let people who want it= compile it from source there, and let them know there isn't any need f= or the ICU add ons.</div><div><br></div><div>An alternative is to go beyond= a Python script and create a full utility in C that does this same work. T= hat would make distribution much easier to all of the platforms. The reason= I did not initially take that route is that Python is so convenient for wo= rking with XML in whereas the library has no such mechanism to readily pars= e it and query in the same way. Obviously it can be done, as osis2mod is al= ready doing that work. Its parsing code could be repurposed to this effect.= </div><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div= ><div><div><br><blockquote type=3D"cite"><div><div dir=3D"ltr"><div class= =3D"gmail_quote"><div><div><br></div><div>So there are pros and cons betwee= n them. I was freshly off of getting the bindings to compile when I wrote t= he first draft of av11n.py so I naturally went that direction. I also try t= o avoid writing parsers when I can leverage existing ones, as grammars can = be notoriously complex to get correct. So that dictated my choices as much = as did anything else, really!</div></div></div></div></div></blockquote><di= v><br></div>My computer science masters degree was in compiler writing! It= =E2=80=99s definitely not for the faint of heart!</div></div></div></blockq= uote><div><br></div><div>Mine was in AI. Also not for the faint of heart, b= ut much more approachable of a consumer product.</div><div>=C2=A0</div><blo= ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left= :1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><br><blockquot= e type=3D"cite"><div><div dir=3D"ltr"><div class=3D"gmail_quote"><div><div>= <br></div><div>Another possible enhancement might be a CLI flag to limit th= e testing range to a particular book (or testament) at a time. I have heard= people talk about having modules split up to one book per file or similar.= If they could say, "Only check this file against Joshua" then it= could keep down a significant amount of extra output. But again - I'm = not really an intended user of it!</div></div></div></div></div></blockquot= e><div><br></div>Great idea. So David=E2=80=99s suggestion of a scope argum= ent.</div></div></div></blockquote><div><br></div><div>Basically, yes.</div= ><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px= 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><d= iv><div><br></div><div>And I=E2=80=99m not an intended user of it either. I= =E2=80=99m just trying to get people to use something other than osis2mod t= o pick a versification. Looking at the Jira issues on osis2mod, in one issu= e a person listed their script that looped over the v11ns and called osis2m= od with each. Yuck!</div></div></div></blockquote><div><br></div><div>Yeah,= using osis2mod in that way seems fraught with trouble. But in the absence = of knowing more about the internal of the library, I can see why someone wo= uld take that approach.</div></div><div class=3D"gmail_quote gmail_quote_co= ntainer"><br></div><div class=3D"gmail_quote gmail_quote_container">--Greg<= /div></div> --000000000000f923b20637f85922-- --===============6647260732657178196== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ sword-devel mailing list: [email protected] http://crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page --===============6647260732657178196==--