Re: Time for a new (minor?) release?

On Tue, 18 Aug 2020 at 20:23, Gary Buhrmaster <[email protected]> wrote:
>
> On Tue, Aug 18, 2020 at 6:54 PM Nick Morrott <[email protected]> wrote:
>
> > Testing tv_grab_fr this evening and should _finally_ push updates later.
>
> As I recall the issue with tv_grab_fr correctly, it
> was that the web pages moved to a responsive
> (dynamic) scrolling approach.  As I suspect
> some of the other sites that some of the grabbers
> scrape will be eventually moving in that direction
> too, are your approaches generalizable for others
> to use for the other scrapers when they need to?

The approach I took for the updated tv_grab_fr source data was to hook
up Firefox's Developer Tools whilst browsing the site and watch the
XHR network accesses stack up.

Once those were understood (and thanks to a sane URL scheme used for
the injected responses) working out how to retrieve the underlying
data (HTML, not JSON) was straightforward.

Not generalizable in this instance, as handling is specific to the
site being accessed.

Cheers,
Nick