Re: Automatic crawling for feed templates: 0watch

On Mon, Jun 27, 2016 at 6:25 AM, Bastian Eicher <[email protected]> wrote:
> Hello all,
>
> I maintain a set of feeds for 3rd party apps at http://0install.de/feeds/
> using 0repo and 0template.
> While updating a feed when a new version of an app is released only takes a
> couple of keystrokes I'd still much prefer if this were a fully automatic
> process.
>
> I was thinking of introducing a new tool to help with this. Going with the
> name "0watch" for now.
> 0watch would take a .xml.template file as input and look for a
> yet-to-be-defined <template> tag in a new XML namespace.
> This tag would point to an HTTP URI along with regular expressions for
> extracting a set of available versions and metadata (e.g. release date) from
> an upstream website.
> Next, 0watch would fetch the current state of the feed the template points
> to in its <feed-for> tag.
> Finally, 0watch would call 0template once for each version found using the
> regular expressions but not currently present in the feed.
>
> Then I could create a simple cronjob like this:
> cd templates
> 0watch templates/*.xml.template
> mv templates/*.xml incoming/
> 0repo
>
> I think this might be best built as a Python tool, pulling in Python
> 0install and 0template as dependencies.
>
> What do you guys think? Would this be a useful tool for you as well?
> Any recommendations for Python libs for parsing/crawling websites?
> Possible alternatives for using RegEx to extract content?

Using regexes could get unwieldy fast - assuming this program is run
only on templates that the author trusts, it might be less painful in
the long run to have a python expression (which could be evaluated in
the scope of some python module which the author maintains on the
side). Then you can offload the complex stuff to a real language,
instead of adding ever more flags an extensions to control the
behaviour.

Generally it sounds like a good idea, although I doubt I'd get any use
of it these days - I haven't been spending the effort required to keep
my feeds up to date, and have found other means (often `nix`) to
fulfil most of my use cases with less effort on my part. Although I
still think ZI feeds are purer, I've found it's not worth the effort I
was putting in (thousands of packages in `nix` that I don't have to
maintain, and new packages are much easier to write/test) and it also
provides a workable solution to distributing compileable source code.
Not to derail the thread, just explaining my rationale.

> Name suggestions?
>
> Regards
> Bastian
>
>
> ------------------------------------------------------------------------------
> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
> Francisco, CA to explore cutting-edge tech and listen to tech luminaries
> present their vision of the future. This family event has something for
> everyone, including kids. Get more information and register today.
> http://sdm.link/attshape
> _______________________________________________
> Zero-install-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/zero-install-devel

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape