Downloading TSV files

[email protected] (Meir Guttman) Wed, 27 Oct 2010 12:48:00 +0200
Newsgroups perl.libwww
Message-ID <!&!AAAAAAAAAAAYAAAAAAAAAKC25OsDcMdGneFIQhy+Pc3CgAAAEAAAAPO9R+mvV9xMmUqWsQ+kkNQBAAAAAA==@guttman.co.il>
Hi everybody!

I am ready to reward the one who will create a script that will do the
following. As for "How much of a reward?" I am not sure, since I have
no clue as to the effort required, so I am open to offers...

The purpose of the script is to automatically download, once a day, a
number of TSV (Tab-Separated-Values) files from the following site:
http://www.tase.co.il/TASEEng/Homepage.htm. The site offers these
files freely, but IMHO makes a conscious effort to block scripts, as
opposed to a human using a standard browser clicking buttons. "Listening"
on the exchange between the browser and the site server using "WireShark"
reveals that the browser is exchanging binary and/or encrypted data
with the server.

One example: to download the full "Trading Info" file, a human using a 
browser could go first to the   
"http://www.tase.co.il/TASEEng/MarketData/Securities/" page. This
brings up an initial page with part of the row and a subset of the
possible columns and with many links to more such partial pages.
To get to a single, full page with a complete set of columns, one needs
to click the "Additional Column" link at the bottom right of the page.
Once the single, all rows, full set of columns page is displayed, the
user can export and download a TSV file by:
    1) clicking the "Export Table Data" bar which reveals a sub bar with
       the choices "Excel 2003 | CSV | TSV "; and
    2) clicking the TSV sub-bar.

A cursory glance at the above URL page source reveals a JavaScript:
    <script language="javascript"> function ShowPrintVersion(strURL)
    {customWindowOpen('/TASEEng/MarketData/Securities/Securities.htm
    '+'?DisplayMode=Print?action=1&SubAction=0','myWindowOne',
    'toolbar=no, menubar=no, location=no, directories=no,
    scrollbars=yes', 660, 500);} </script>
so maybe the whole issue is how to integrate and execute JavaScripts
in LWP?

I have also to warn you that this site is notoriously finicky. It is
probably overloaded (and overwhelmed) by users. Lately I experience
many types of false downloads and other problems that require repeated
downloading trials. (I'll supply the file integrity check.)

Would one of you folks pick up this challenge?

Regards,
Meir