Re: Automatic crawling for feed templates: 0watch

On 26 June 2016 at 21:25, Bastian Eicher <[email protected]> wrote:
> Hello all,
>
> I maintain a set of feeds for 3rd party apps at http://0install.de/feeds/
> using 0repo and 0template.
> While updating a feed when a new version of an app is released only takes a
> couple of keystrokes I'd still much prefer if this were a fully automatic
> process.
>
> I was thinking of introducing a new tool to help with this. Going with the
> name "0watch" for now.
> 0watch would take a .xml.template file as input and look for a
> yet-to-be-defined <template> tag in a new XML namespace.
> This tag would point to an HTTP URI along with regular expressions for
> extracting a set of available versions and metadata (e.g. release date) from
> an upstream website.
> Next, 0watch would fetch the current state of the feed the template points
> to in its <feed-for> tag.
> Finally, 0watch would call 0template once for each version found using the
> regular expressions but not currently present in the feed.
>
> Then I could create a simple cronjob like this:
> cd templates
> 0watch templates/*.xml.template
> mv templates/*.xml incoming/
> 0repo
>
> I think this might be best built as a Python tool, pulling in Python
> 0install and 0template as dependencies.
>
> What do you guys think? Would this be a useful tool for you as well?
> Any recommendations for Python libs for parsing/crawling websites?
> Possible alternatives for using RegEx to extract content?
> Name suggestions?

Extra automation is always good. Not sure about the cron job, though.
I feel there should be a human in the loop running the new version at
least once before publishing it.


-- 
Dr Thomas Leonard        http://roscidus.com/blog/
GPG: DA98 25AE CAD0 8975 7CDA  BD8E 0713 3F96 CA74 D8BA

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape