Re: cache a object in modperl

On Sun, Sep 13, 2020, at 21:51, Wesley Peng wrote:
> For work requirement, I need to access IANA TLD database.
> 
> So I wrote this perl module:
> https://metacpan.org/pod/Net::IANA::TLD
> 
> But, for each new() in the module, the database file will be downloaded 
> from IANA's website.
> 
> I know this is pretty Inefficient.

Not only inefficient but you abuse remote resources and you risk having
your access being rate limited or just blocked.

You should use caching features available by HTTP as the resource has an ETag:

$ wget -SqO /dev/null http://www.internic.net/domain/root.zone
  HTTP/1.1 200 OK
  Date: Mon, 14 Sep 2020 15:17:50 GMT
  Server: Apache
  Last-Modified: Mon, 14 Sep 2020 05:44:00 GMT
  Content-Length: 2164237
  Vary: Accept-Encoding
  ETag: "21060d-5af3f856f0800"
  Accept-Ranges: bytes
  Cache-Control: max-age=420
  Expires: Mon, 14 Sep 2020 15:22:04 GMT
  X-Frame-Options: SAMEORIGIN
  Referrer-Policy: origin-when-cross-origin
  Content-Security-Policy: upgrade-insecure-requests
  Age: 165
  Keep-Alive: timeout=2, max=358
  Connection: Keep-Alive
  Content-Type: text/plain; charset=UTF-8
  Content-Language: en


So you can do a conditional GET as long as you store the latest ETag
on your side:

$ wget -SqO /dev/null --header 'If-None-Match: "21060d-5af3f856f0800"' http://www.internic.net/domain/root.zone
  HTTP/1.1 304 Not Modified
  Date: Mon, 14 Sep 2020 15:20:43 GMT
  Server: Apache
  Connection: Keep-Alive
  Keep-Alive: timeout=2, max=358
  ETag: "21060d-5af3f856f0800"
  Expires: Mon, 14 Sep 2020 15:22:04 GMT
  Cache-Control: max-age=420
  Vary: Accept-Encoding


All of this has nothing to do with modperl and very lightly to do with Perl at all in fact.

See also the "Cache-Control" and "Age" headers.

Your module on CPAN should take care of that automatically.

PS: TLDs do not vary so much, fetching once per day or once per week should be enough (with manual exceptional override for those cases that need it). But it depends why you do it. Note that the whole content is also available as a zone transfer from various root servers.

-- 
  Patrick Mevzek