Re: Requesting URLs containing German umlauts

On 08/16/2012 10:18 AM, Pawel Krol wrote:
> Hello!
>
> I would like to ask you for a little assistance with the following issue...
>
> There are websites, which contain special German characters in their
> URLs, for example: http://www.pbb-planungsbĂźro-bartsch.de (known as
> "umlauts").
>
> I have been unsuccessfully trying to retrieve contents of such
> websites using Perl (basically the purpose of it is to check, whether
> the URL is valid/invalid - maybe there's a simpler way to do it?).
>
> Here's a code snippet, which you can try out immediately:
>
> #!/opt/local/bin/perl
>
> use Data::Dumper;
> use LWP::UserAgent;
>
> my $url = q{http://www.pbb-planungsbĂźro-bartsch.de};
> # my $url = q{http://www.pbb-planungsb%C3%BCro-bartsch.de/};
>
> my $ua = LWP::UserAgent->new;
> my $response = $ua->get($url);
>
> warn Dumper $response;
>
> __END__

The critical piece of data you need is "IRI" - Internatinoalized 
Resource Identifier - HTTP the protocol requires that the HOST part of 
the URI be a valid Domain Name System identifier, which is limited to 
the characters A-Z, a-z, 0-9, and hyphen (See RFC 1035, Sec 2.3.1) 
(Curiously - RFC 2181 states rather emphatically that any octet stream 
must be acceptable as a resource label ... )

Anyway -- there is a well accepted technique for mapping strings like 
gĂźero to a name that fits into the "LDH" (letters, digits, hyphen) rule 
- and it's called "internationalized domain name"

Read all about it in 
http://search.cpan.org/~cfaerber/Net-IDN-Encode-2.003/lib/Net/IDN/Standards.pod