Re: HTTPS remote SWORD repo access
Tobias Klein <[email protected]>
| Newsgroups | gmane.comp.literature.sword.devel |
|---|---|
| Message-ID | <[email protected]> |
Hi Greg, when configuring SWORD for my Android build (based on CMake) I currently do not see CURL being detected automatically and therefore there is no CURL in my Android SWORD build: -- Configuring your system to build libsword. -- SWORD Version 1009000000 -- -- SEARCHING FOR SYTEM PACKAGES -- System regex.h: Yes -- -- CONFIGURING SOURCE LIST -- ZLib: system /opt/Android/SDK/ndk/r27c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libz.so -- bzip2: no -- xz: no *-- cURL: no* -- CLucene: no -- PkgConfig: yes -- ICU: no -- Regex.h: system /opt/Android/SDK/ndk/r27c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include -- Building Static library. -- Setting link libraries to /opt/Android/SDK/ndk/r27c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/21/libz.so Best regards, Tobias On 2/28/25 18:25, Greg Hellings wrote: > This is all wonderful, but it's really just efficiency improvements. > I've been testing locally with nginx, and the only setting I need is > to set `autoindex on;` in the location that points to the Sword repo. > Everything works fine, then. I have tested in the past with Apache > (although it has been a very long time since I've touched it) and it > works fine there. > > There's nothing currently stopping us from swapping to HTTP or HTTPS > client-based access anywhere that is built against curl. Which is > every Linux distro I've seen, MacOS CLI distributions, and the Xiphos > Windows builds. Also, I'm pretty sure all the known mobile apps > support it, as well. > > --Greg > > On Fri, Feb 28, 2025 at 10:05 AM Troy A. Griffitts > <[email protected]> wrote: > > Hey guys, > > We did some work in the past to make this transition begin to > happen automatically without user actions. I think there was an > outline in here a year or so back. Basically, the path forward was: > > 1) to standardize a location in a SWORD repository for .zip > package. This has many benefits beyond helping us get to HTTPS > everywhere. It conceptually allows reading directly from the zip > file by the engine. It could facilitate the engine itself > building the zip packages from an exploded tree. All transports, > including the dominant FTP transport right now could look to see > if there is an available single .zip package for download instead > of downloading each individual file for a module during a remote > install, and I am sure more ideas people will imagine in the > future... but simply this step was to standardize the location of > .zip package which each represent a zip of a single package based > at the root of a sword module repo, e.g., > > KJV.zip: > mods.d/kjv.conf > modules/texts/ztext/kjv/nt.bzs > modules/texts/ztext/kjv/ot.bzs > modules/texts/ztext/kjv/ot.bzv > modules/texts/ztext/kjv/nt.bzv > modules/texts/ztext/kjv/ot.bzz > modules/texts/ztext/kjv/nt.bzz > > This location decided on was pretty straighforward, at the root of > the sword module repo, packages/ > > You'll notice this if you have a look at the crosswire repo here: > > https://crosswire.org/ftpmirror/pub/sword/raw/ > > The latest version of the engine does use this now and tries to > downoad a package if available, before it falls back to the > previous behavior of downloading each individual file. > > 2) Support 'chained' transport providers in the engine, allowing > multiple transports to the same repository. This has been > completed and the first use of this concept is the recognition of > a new HTTPSPackagePreference option in installmgr's configuration > file. > > The value of the property is: existing repo > name|hostname|repository_path > > You can see this added for the CrossWire repository in the master > repo list registry at: > > https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf > > HTTPSPackagePreference=CrossWire|crosswire.org > <http://crosswire.org>|/ftpmirror/pub/sword/raw > FTPSource=CrossWire|ftp.crosswire.org > <http://ftp.crosswire.org>|/pub/sword/raw > > This tells installmgr that if a user requests a package install > from "CrossWire" then chain 2 transport provider attempts together > and prefer the HTTPS transport first. > > > The hope is that this path forward will see the vast amount of > SWORD frontends begin to start using HTTPS instead of using FTP, > to facilitate a smooth transition from FTP to HTTPS. The idea is > that if a repo owner sets everything up correctly, and a user has > the latest software and latest configuration information from our > registry of known SWORD repos ( > https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf ), > everything will flow over HTTPS instead of FTP and once we see FTP > traffic drop to very low levels, a repo owner could choose to > discontinue support for FTP if they desire. > > Michael, I am glad you have your server back up and running well > again. Thank you for all that you do to distribute the Word of > God to a needing world. > > Troy > > > On 2/27/25 9:45 PM, Michael Johnson wrote: >> >> Hi David, >> >> Yes, under eBible.org, where it says "HTTP access", it should say >> "HTTPS and HTTP access". Also, any repository that also supports >> HTTPS and/or HTTP access should explicitly say so. As Greg >> mentioned, mobile often blocks FTP. FTP may not be dead, yet, but >> it is in hospice care. HTTP likewise can cause problems with >> mobile platforms, where the major app stores now require use of >> HTTPS instead of insecure HTTP in most cases. I still support >> insecure HTTP connections to eBible.org except for at >> shop.eBible.org <http://shop.eBible.org> and >> https://eBible.org/cgi-bin/contact.cgi >> <https://eBible.org/cgi-bin/contact.cgi> but there may come a day >> when HTTP goes away, too. >> >> On 2/26/25 22:03, David Haslam wrote: >>> Hi Michael, >>> >>> /Does anything need to be changed here?/ >>> Official and Affiliated Module Repositories - CrossWire Bible >>> Society >>> <https://wiki.crosswire.org/Official_and_Affiliated_Module_Repositories#eBible.org> >>> >>> _Everyone_: Except for the note under *AndBibleExtra*, there's >>> no mention of *https* ! >>> /Do we need to make any wider changes to be future proof?/ >>> >>> Best regards, >>> >>> David >>> >>> Sent with Proton Mail <https://proton.me/mail/home> secure email. >>> >>> On Thursday, February 27th, 2025 at 7:13 AM, Michael Johnson >>> <[email protected]> <mailto:[email protected]> wrote: >>>> >>>> >>>> On 2/26/25 17:12, Greg Hellings wrote: >>>>> >>>>> >>>>> On Wed, Feb 26, 2025, 7:33 PM Kahunapule Michael Johnson >>>>> <[email protected]> wrote: >>>>> >>>>> Greetings from Maui! >>>>> >>>>> tldr: upgrade your Sword apps to always use https instead >>>>> of http or ftp to access repositories ASAP. >>>>> >>>>> >>>>> While technically any network acess other than anonymous FTP >>>>> support is optionally supported only with a build dep, in >>>>> reality there is no need to support anything other than HTTPS. >>>>> Every Linux distribution, and Windows build of note has >>>>> libcurl, the Brew version is also built against it, and the >>>>> HTTP(S) support was added because mobile often blocks FTP. >>>>> >>>>> So you're basically completely safe. >>>> >>>> Awesome! >>>> >>>> >>>>> >>>>> >>>>> As many of you are probably aware, the last week was not a >>>>> model of reliability for the eBible.org repository, or for >>>>> the rest of the eBible.org site. On the 19th of February, >>>>> the eBible.org server hardware failed. Exactly what >>>>> failure, I don't know, because it was in a data center >>>>> over 4,000 miles from my house. I just knew that it >>>>> wouldn't talk to me in any of the 3 ways I can normally >>>>> access the leased dedicated server. No worries, because I >>>>> have a fast backup, right? I allocated a new dedicated server >>>>> from the same company (Ionos) and attempted to restore >>>>> from a backup. That failed with about 80 error messages. >>>>> Next plan: restore from a mirror image of the server in my >>>>> home office. That actually worked, but it took more than 3 >>>>> days to get all of the data there (about 300 GBytes), plus >>>>> time to get all of the configuration right. In the mean >>>>> time, my other leased server (the one that didn't crash, >>>>> hosting 24 other sites) gave early warning signs that it >>>>> was not going to be in service much longer. Then >>>>> everything worked except that I forgot a couple of tweaks >>>>> I had to do to make the ftp server compatible with Sword. >>>>> I fixed that, and things were still not OK. EBible.org >>>>> availability kept going up and down like a yo-yo, mostly >>>>> because the remote control software I was using was not >>>>> designed to handle multiple IP addresses per server and >>>>> anonymous ftp sites. Also, the cost of allocating multiple >>>>> IP v4 addresses has gone up. Anonymous ftp is pretty much >>>>> obsolete. I will be dropping it, but slowly. >>>>> >>>>> >>>>> A Herculean effort, but I'm glad for you that your recovery >>>>> was successful! I'm curious why you need 4 separate addresses? >>>>> What is the need, there? >>>> >>>> So far, I have been using Plesk to set up virtual hosts. I have >>>> 25 sites (and some aliases for those), some of which are much >>>> more important than others. Plesk lets me share one IP address >>>> with all sites except any site that has an anonymous ftp >>>> service associated with it. The only site I have that has an >>>> anonymous ftp service associated with it, of course, is the >>>> ftp.eBible.org <http://ftp.eBible.org> Sword repository. So I >>>> had to assign 2 IP version 4 addresses to the server. For a >>>> long time, I was running 2 servers with every site on them for >>>> redundancy. I had stopped doing that because the sites grew too >>>> large for one of the servers I was renting, and I thought I had >>>> a workable fast backup/restore plan, unlike when I had >>>> extremely slow and expensive Internet in Papua New Guinea. (I >>>> have some serious space in audio and video Bibles.) So that is >>>> 2 servers x 2 IP addresses = 4 IP addresses. But that >>>> configuration was unstable, so I went to just one IP address >>>> per server by fighting my old ally, Plesk, using manual ProFTP >>>> configuration (and a cron job to slap my configuration back >>>> whenever Plesk rewrites it). That is not a really good >>>> long-term solution, though. >>>> >>>>> ... >>>>> Would you like a hand building up some DR or deployment >>>>> automation so you can avoid needing to remember settings? IT >>>>> automation is one of my primary skillsets, so if you'd like >>>>> any sort of help setting it up, let me know. For instance, >>>>> it's not too hard to put together automation scripts to run on >>>>> a provisioned box to stand up the web server, ftp server, etc >>>>> so that you don't need to manually edit files and the like. >>>> That would be useful. That could be a way to escape my >>>> dependence on and fight with Plesk. >>>>> >>>>> Alternatively, have you considered an alternative way to host >>>>> the data? You could probably build a Container image with all >>>>> the files in it and host that on something like Amazon >>>>> Container Service or any of the many cloud Kubernetes hosts >>>>> around. A container image would also make it easy for someone >>>>> to grab the whole collection and make it available in an >>>>> offline context the way they can with the old CD images Troy >>>>> used to distribute. >>>> I have looked at alternatives in the past, but it may be worth >>>> looking again. When I last looked, AWS was more expensive at my >>>> traffic levels and site counts than using a rented dedicated >>>> server. Another alternative might be hosting at my house when >>>> (if?) Hawaiian Telephone makes good on its promise to bring >>>> fiber Internet to my neighborhood. (It is actually available >>>> about a half mile away, right now, but I haven't seen them >>>> working on it around here.) >>>>> >>>>> Or even put the files into an object storage container if >>>>> you're dedicated to eliminating FTP access eventually. With >>>>> just a small shell script you can push the needed files and >>>>> their indexes into an S3, Ceph, etc object storage service and >>>>> then you wouldn't need to run a dedicated server with them to >>>>> manage uptime. All of those services offer ways to expose the >>>>> files over HTTPS. >>>>> >>>>> As I said on Facebook, I'm happy to lend a hand if there's >>>>> anything I can do to help smooth your infrastructure! I can >>>>> even host an emergency mirror if need be, as I have pretty >>>>> reliable Internet and electric when my neighbors don't drive >>>>> into the electric poles. This year I'm dedicating some of my >>>>> time to working on home electric backups! >>>> >>>> Thank you, Greg. I may take you up on that... >>>> >>>> -- >>>> >>>> Peace, >>>> */Michael Johnson/** >>>> 26 HIWALANI LOOP • MAKAWAO HI 96768-8747*• USA >>>> mljohnson.org <https://mljohnson.org/> • eBible.org >>>> <https://eBible.org> • WorldEnglish.Bible >>>> <https://WorldEnglish.Bible> • PNG.Bible <https://PNG.Bible> >>>> Signal/Telegram/WhatsApp/Telephone: +1 808-333-6921 >>>> Skype: kahunapule • Telegram: @kahunapule • Facebook: >>>> fb.me/kahunapule <https://www.facebook.com/kahunapule> >>>> >>> >>> >>> _______________________________________________ >>> sword-devel mailing list:[email protected] >>> http://crosswire.org/mailman/listinfo/sword-devel >>> Instructions to unsubscribe/change your settings at above page >> -- >> >> Peace, >> */Michael Johnson/** >> 26 HIWALANI LOOP • MAKAWAO HI 96768-8747*• USA >> mljohnson.org <https://mljohnson.org/> • eBible.org >> <https://eBible.org> • WorldEnglish.Bible >> <https://WorldEnglish.Bible> • PNG.Bible <https://PNG.Bible> >> Signal/Telegram/WhatsApp/Telephone: +1 808-333-6921 >> Skype: kahunapule • Telegram: @kahunapule • Facebook: >> fb.me/kahunapule <https://www.facebook.com/kahunapule> >> >> >> _______________________________________________ >> sword-devel mailing list:[email protected] >> http://crosswire.org/mailman/listinfo/sword-devel >> Instructions to unsubscribe/change your settings at above page > _______________________________________________ > sword-devel mailing list: [email protected] > http://crosswire.org/mailman/listinfo/sword-devel > Instructions to unsubscribe/change your settings at above page > > > _______________________________________________ > sword-devel mailing list:[email protected] > http://crosswire.org/mailman/listinfo/sword-devel > Instructions to unsubscribe/change your settings at above page _______________________________________________ sword-devel mailing list: [email protected] http://crosswire.org/mailman/listinfo/sword-devel Instructions to unsubscribe/change your settings at above page