fetching files with javascript

	npr.org started requiring javascript to access their HTML
files (but not their audio files); gocomics.com did a year ago.  I
wrote a short script that fetches them with nodejs.  This works well
for me for npr.org and gocomics.com because I fetch the home page of
'All Things Considered' (for example), from that page I extract the
URIs of all the segments for the day; I can fetch the segments via
normal HTTP.  I don't want to browse either.
	I thought of adding this as a function to lynx: press a key on
a highlighted link to fetch it with node's library into the temp
directory, then deal with it normally.
	Is this of value to anyone else?  I figure if npr.org's doing
it...

russell bell


const args = process.argv.slice(2); // Get arguments after the script name
const argument = args[0]; // Access the first argument passed
console.log(`Argument passed: ${argument}`);

async function fetchWebPage(url) {
  try {
    const response = await fetch(url,  { headers: { "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 lynx" } });
    if (response.ok) {
      const data = await response.text();
      console.log(data);
    } else {
      console.error(`Failed to fetch the page. Status: ${response.status}`);
    }
  } catch (error) {
    console.error('Error fetching the page:', error);
  }
}

fetchWebPage(`${argument}`);


Usage example:  node $HOME/bin/javafetch.js https://www.gocomics.com/super-fun-pak-comix > super-fun-pak-comix.html

Note on the script: gocomics.com stopped working earlier this week.  I
had to pass a non-default user-agent.  If I put this in lynx it will
use lynx's.  If you want to use the script yourself you can modify the
user-agent.