xmltv/grab/uk_rt tv_grab_uk_rt,1.23,1.24
Nick Morrott <[email protected]>
| Newsgroups | gmane.comp.tv.xmltv.cvs |
|---|---|
| Message-ID | <[email protected]> |
Update of /cvsroot/xmltv/xmltv/grab/uk_rt
In directory vz-cvs-3.sog:/tmp/cvs-serv25867
Modified Files:
tv_grab_uk_rt
Log Message:
Instead of disabling the automated UTF-8 fixups when fixups are disabled at config time, only disable those manual fixups which are sourced from the utf8-fixups supplemental file. Removing non-printing/control/malformed characters will always be required for optimal output
Index: tv_grab_uk_rt
===================================================================
RCS file: /cvsroot/xmltv/xmltv/grab/uk_rt/tv_grab_uk_rt,v
retrieving revision 1.23
retrieving revision 1.24
diff -C2 -d -r1.23 -r1.24
*** tv_grab_uk_rt 10 Jun 2012 08:04:15 -0000 1.23
--- tv_grab_uk_rt 10 Jun 2012 08:43:26 -0000 1.24
***************
*** 25,31 ****
##############################################
- # - split out utf8 fixups into these which should always be applied
- # and those which are optional (like misencoded chars)
- #
# - wrap date/file tests in evals if there is a risk of failure
#
--- 25,28 ----
***************
*** 1001,1007 ****
# detect/correct UTF-8 errors in source data
! if ($utf8_fixups_status eq 'enabled') {
! $page = process_utf8_fixups($page, $rt_name, $rt_listings_uri);
! }
# Decode source UTF-8 octets, process for HTML entities, and encode
--- 998,1002 ----
# detect/correct UTF-8 errors in source data
! $page = process_utf8_fixups($page, $rt_name, $rt_listings_uri);
# Decode source UTF-8 octets, process for HTML entities, and encode
***************
*** 1939,2017 ****
# without needing to update the grabber itself.
#
! # Unicode Replacement Character (U+FFFD)
! # ======================================
! #
! # The UTF-8 source data may also contain the bytes [EF][BF][BD] which
! # are the UTF-8 encoding of the Unicode Replacement Character U+FFFD.
! # It is likely that these are introduced during preparation of the
! # listings data by the Radio Times, as any characters that cannot be
! # understood are replaced by this character.
! #
! t(" Looking for Unicode Replacement Character...");
! if (m/\xEF\xBF\xBD/) {
! if (%utf8_fixups && exists $utf8_fixups{'EFBFBD'}) {
! foreach my $bad_chars (keys %{$utf8_fixups{'EFBFBD'}}) {
! my $replacement = $utf8_fixups{'EFBFBD'}{$bad_chars};
! # Search for the regex string and replace with char string
! if ($page =~ s/$bad_chars/$replacement/g) {
! t(" Replaced Unicode Replacement Character with \""
! . $replacement . "\"");
}
}
}
- if ($page =~ s/\xEF\xBF\xBD/\x3F/g) {
- t(" After fixups, data for '$rt_name' still contains Unicode "
- . "Replacement character. Replaced with \"?\"\n");
- $hadEFBFBD{$rt_name} = $rt_listings_uri;
- }
- }
! # Double-encoded Unicode Replacement Character (6 bytes)
! # ======================================================
! #
! # The correct encoding for the Unicode Replacement Character is
! # [EF][BF][BD], however it has been seen double-encoded in the listings
! # data as [C3][AF][C2][BF][C2][BD]. As with the normal replacement
! # character, there is no way to determine which replacement character
! # to use in this case, so we substitute a '?' char if we cannot handle
! # the specific occurence. This error needs to have been seen at least
! # once in source data to be able to construct a suitable fixup.
! #
! t(" Looking for double-encoded Unicode Replacement Character...");
! if (m/\xC3\xAF\xC2\xBF\xC2\xBD/) {
! if (%utf8_fixups && exists $utf8_fixups{'C3AFC2BFC2BD'}) {
! foreach my $bad_chars (keys %{$utf8_fixups{'C3AFC2BFC2BD'}}) {
! my $replacement = $utf8_fixups{'C3AFC2BFC2BD'}{$bad_chars};
! # Search for the regex string and replace with char string
! if ($page =~ s/$bad_chars/$replacement/g) {
! t(" Replaced double-encoded Unicode Replacement Character with \""
! . $replacement . "\"");
}
}
}
- if ($page =~ s/\xC3\xAF\xC2\xBF\xC2\xBD/\x3F/g) {
- t(" After fixups, data for '$rt_name' still contains "
- . "double-encoded Unicode Replacement character. "
- . "Replaced with \"?\"\n");
- $hadC3AFC2BFC2BD{$rt_name} = $rt_listings_uri;
- }
- }
! # Mis-encoded characters in range [C2][80-9F]
! # ===========================================
! #
! # Single characters that are seen in the source data as bytes in the
! # range [C2][80-9F] that UTF-8 decode as non-printing characters
! # instead of their intended character.
! #
! t(" Looking for mis-encoded characters in range [C2][80-9F]...");
! if (m/\xC2[\x80-\x9F]/) {
! if (%utf8_fixups && exists $utf8_fixups{'C2809F'}) {
! foreach my $bad_chars (keys %{$utf8_fixups{'C2809F'}}) {
! my $replacement = $utf8_fixups{'C2809F'}{$bad_chars};
! # Search for the regex string and replace with char string
! if ($page =~ s/$bad_chars/$replacement/g) {
! t(" Replaced mis-encoded characters \"" . $bad_chars
! . "\" with \"". $replacement . "\"");
}
}
--- 1934,2015 ----
# without needing to update the grabber itself.
#
! if ($utf8_fixups_status eq 'enabled') {
!
! # Unicode Replacement Character (U+FFFD)
! # ======================================
! #
! # The UTF-8 source data may also contain the bytes [EF][BF][BD] which
! # are the UTF-8 encoding of the Unicode Replacement Character U+FFFD.
! # It is likely that these are introduced during preparation of the
! # listings data by the Radio Times, as any characters that cannot be
! # understood are replaced by this character.
! #
! t(" Looking for Unicode Replacement Character...");
! if (m/\xEF\xBF\xBD/) {
! if (%utf8_fixups && exists $utf8_fixups{'EFBFBD'}) {
! foreach my $bad_chars (keys %{$utf8_fixups{'EFBFBD'}}) {
! my $replacement = $utf8_fixups{'EFBFBD'}{$bad_chars};
! # Search for the regex string and replace with char string
! if ($page =~ s/$bad_chars/$replacement/g) {
! t(" Replaced Unicode Replacement Character with \""
! . $replacement . "\"");
! }
}
}
+ if ($page =~ s/\xEF\xBF\xBD/\x3F/g) {
+ t(" After fixups, data for '$rt_name' still contains Unicode "
+ . "Replacement character. Replaced with \"?\"\n");
+ $hadEFBFBD{$rt_name} = $rt_listings_uri;
+ }
}
! # Double-encoded Unicode Replacement Character (6 bytes)
! # ======================================================
! #
! # The correct encoding for the Unicode Replacement Character is
! # [EF][BF][BD], however it has been seen double-encoded in the listings
! # data as [C3][AF][C2][BF][C2][BD]. As with the normal replacement
! # character, there is no way to determine which replacement character
! # to use in this case, so we substitute a '?' char if we cannot handle
! # the specific occurence. This error needs to have been seen at least
! # once in source data to be able to construct a suitable fixup.
! #
! t(" Looking for double-encoded Unicode Replacement Character...");
! if (m/\xC3\xAF\xC2\xBF\xC2\xBD/) {
! if (%utf8_fixups && exists $utf8_fixups{'C3AFC2BFC2BD'}) {
! foreach my $bad_chars (keys %{$utf8_fixups{'C3AFC2BFC2BD'}}) {
! my $replacement = $utf8_fixups{'C3AFC2BFC2BD'}{$bad_chars};
! # Search for the regex string and replace with char string
! if ($page =~ s/$bad_chars/$replacement/g) {
! t(" Replaced double-encoded Unicode Replacement Character with \""
! . $replacement . "\"");
! }
}
}
+ if ($page =~ s/\xC3\xAF\xC2\xBF\xC2\xBD/\x3F/g) {
+ t(" After fixups, data for '$rt_name' still contains "
+ . "double-encoded Unicode Replacement character. "
+ . "Replaced with \"?\"\n");
+ $hadC3AFC2BFC2BD{$rt_name} = $rt_listings_uri;
+ }
}
! # Mis-encoded characters in range [C2][80-9F]
! # ===========================================
! #
! # Single characters that are seen in the source data as bytes in the
! # range [C2][80-9F] that UTF-8 decode as non-printing characters
! # instead of their intended character.
! #
! t(" Looking for mis-encoded characters in range [C2][80-9F]...");
! if (m/\xC2[\x80-\x9F]/) {
! if (%utf8_fixups && exists $utf8_fixups{'C2809F'}) {
! foreach my $bad_chars (keys %{$utf8_fixups{'C2809F'}}) {
! my $replacement = $utf8_fixups{'C2809F'}{$bad_chars};
! # Search for the regex string and replace with char string
! if ($page =~ s/$bad_chars/$replacement/g) {
! t(" Replaced mis-encoded characters \"" . $bad_chars
! . "\" with \"". $replacement . "\"");
! }
}
}
***************
*** 2019,2022 ****
--- 2017,2023 ----
}
+ # With manual replacements handled above, finally remove any
+ # remaining bad/non-printing characters we find
+
# Replacements for specific strings seen in source data
# =====================================================
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/