Re: Komodo 6 - find in files in utf8 without bom
Trent Mick <[email protected]> Sat, 6 Nov 2010 21:56:43 -0700
| Newsgroups | gmane.comp.ide.komodo.general |
|---|---|
| Message-ID | <[email protected]> |
Have a fix for it. The "problem" only happens for Perl files because
Perl has a bone-headed default encoding: iso8859-1. :)
Anyway, the problem is this:
"textinfo.py"'s encoding detection heuristics will attempt a
filetype's default encoding (in this case Perl's iso8859-1) before
getting to the attempt at using UTF-8. For languages like XML (default
UTF-8) and Python (default ASCII) the right thing happens with a file
like the one I showed below. The problem is that a single-byte
encoding like iso8859-1 always *appears* to work -- where "work" means
that Python's `str.decode(ENCODING)` doesn't raise an exception.
Solution is to move the "try the language's default encoding, if it
has one" step in the encoding detection heuristics later in the
process. A slightly better fix might be:
- to only do this for those "single-byte encodings" that always "work"; and/or
- move this heuristic step down *further* below the "chardet" step
(see attached patch)
But I don't think either of those "better" fixes are worth the bother.
Patch attached.
Karel,
BTW, a quick hack fix is to remove the "default encoding" for Perl in
your Komodo install something like this:
{{{
Index: langinfo_prog.py
===================================================================
--- langinfo_prog.py (revision 6654)
+++ langinfo_prog.py (working copy)
@@ -69,21 +69,21 @@
class PerlLangInfo(LangInfo):
name = "Perl"
conforms_to_bases = ["Text"]
exts = ['.pl', '.pm', '.t']
magic_numbers = [
(0, "regex", re.compile(r'\A#!.*perl.*$', re.I | re.M)),
]
filename_patterns = ["Construct", "Conscript"] # Cons
make-replacement tool files
# http://search.cpan.org/~rgarcia/encoding-source-0.02/lib/encoding/source.pm
- default_encoding = "iso8859-1"
+ #default_encoding = "iso8859-1"
# Perl >= 5.8.0
# http://perldoc.perl.org/encoding.html
# use encoding "<encoding-name>";
# "Somewhat broken."
# Perl >= 5.9.5
# http://search.cpan.org/~rgarcia/encoding-source-0.02/lib/encoding/source.pm
# use encoding::source "<encoding-name>";
# "This is like the encoding pragma, but done right."
encoding_decl_pattern = re.compile(
r"""use\s+encoding(?:::source)?\s+(['"])(?P<encoding>[\w-]+)\1""")
}}}
2010/11/6 Trent Mick <[email protected]>:
> "textinfo.py" (used by "Find in Files") is incorrectly guessing the
> encoding of this file. As a result, "Find/Replace in Files" will mess
> up for non-ASCII portions of the file.
>
> Eric,
> Perhaps you could open a bug on textinfo.py?
>
> -----
> $ cat findit.pl
> I created a file containing the text "Vyváženě prožitý čas přeji",
> set the encoding to utf-8, and could find the text
> when the file was saved both with and without a BOM.
> [20:56:47 trentm@banana:~/tmp/f]
> $ file findit.pl
> findit.pl: UTF-8 Unicode English text
> [20:56:51 trentm@banana:~/tmp/f]
> $ python ~/as/komodo/src/python-sitelib/textinfo.py findit.pl
> findit.pl: Perl, iso8859-1
> ----
>
>
> Trent
>
> --
> Trent Mick
>
--
Trent Mick
_______________________________________________
Komodo-discuss mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Other options: http://listserv.ActiveState.com/mailman/listinfo/Komodo-discuss
textinfo-perl.patch
(application/octet-stream, 6.8 KB)
Index: textinfo.py
===================================================================
--- textinfo.py (revision 6654)
+++ textinfo.py (working copy)
@@ -41,21 +41,21 @@
_classify_encoding for details), accurately identifies binary files, and
provides detailed meta information of text files.
>>> import textinfo
>>> path = __file__
>>> if path.endswith(".pyc"): path = path[:-1]
>>> ti = textinfo.textinfo_from_path(path)
>>> ti.__class__
<class 'textinfo.TextInfo'>
>>> ti.encoding
- 'ascii'
+ 'utf-8'
>>> ti.file_type_name
'regular file'
>>> ti.is_text
True
>>> ti.lang
'Python'
>>> ti.langinfo
<Python LangInfo>
...plus a number of other useful information gleaned from the file. To see
@@ -468,28 +468,28 @@
4. Lang-specific (if we know the lang already):
* if this is Python, look for coding: decl and try that
* if this is Perl, look for use encoding decl and try that
* ...
5. XML: According to the XML spec the rule is the XML prolog
specifies the encoding, or it is UTF-8.
6. HTML: Attempt to use Content-Type meta tag. Try the given
charset, if any.
7. Emacs-style "coding" local var.
8. Vi[m]-style "fileencoding" local var.
- 9. Lang-specific fallback. E.g., UTF-8 for XML, ascii for Python.
- 10. Heuristic checks for UTF-16 without BOM.
- 11. Give UTF-8 a try, it is a pretty common fallback.
+ 9. Heuristic checks for UTF-16 without BOM.
+ 10. Give UTF-8 a try, it is a pretty common fallback.
We must do this before a possible 8-bit
`locale.getpreferredencoding()` because any UTF-8 encoded
document will decode with an 8-bit encoding (i.e. will decode,
just with bogus characters).
- 12. chardet (http://chardet.feedparser.org/)
+ 11. Lang-specific fallback. E.g., UTF-8 for XML, ascii for Python.
+ 12. chardet (http://chardet.feedparser.org/), if CHARDET_ENABLED == True
13. locale.getpreferredencoding()
14. iso8859-1 (in case `locale.getpreferredencoding()` is UTF-8
we must have an 8-bit encoding attempt).
TODO: Is there a worry for a lot of false-positives for
binary files.
Notes:
- A la Universal Feed Parser, if some
supposed-to-be-authoritative encoding indicator is wrong (e.g.
the BOM, the Python 'coding:' decl for Python),
@@ -640,40 +640,21 @@
norm_vi_encoding)
self.encoding = norm_vi_encoding
return
else:
log.debug("encoding: Vi[m] coding var (%r) was *wrong*",
norm_vi_encoding)
self._encoding_bozo(
u"Vi[m] coding var (%s) could not decode %s"
% (norm_vi_encoding, self._accessor))
- # 9. Lang-specific fallback (e.g. XML -> utf-8, Python -> ascii, ...).
- fallback_encoding = None
- fallback_lang = None
- if self.langinfo:
- fallback_lang = self.langinfo.name
- fallback_encoding = self.langinfo.conformant_attr("default_encoding")
- if fallback_encoding:
- if self._accessor.decode(fallback_encoding):
- log.debug("encoding: fallback encoding for %s: %r",
- fallback_lang, fallback_encoding)
- self.encoding = fallback_encoding
- return
- else:
- log.debug("encoding: %s fallback encoding (%r) was *wrong*",
- fallback_lang, fallback_encoding)
- self._encoding_bozo(
- u"%s fallback encoding (%s) could not decode %s"
- % (fallback_lang, fallback_encoding, self._accessor))
-
- # 10. Heuristic checks for UTF-16 without BOM.
+ # 9. Heuristic checks for UTF-16 without BOM.
utf16_encoding = None
head_odd_bytes = head_bytes[0::2]
head_even_bytes = head_bytes[1::2]
head_markers = ["<?xml", "#!"]
for head_marker in head_markers:
length = len(head_marker)
if head_odd_bytes.startswith(head_marker) \
and head_even_bytes[0:length] == '\x00'*length:
utf16_encoding = "utf-16-le"
break
@@ -697,27 +678,49 @@
pass
else:
if head_odd_bytes[idx:idx+length] == '\x00'*length:
utf16_encoding = "utf-16-be"
if utf16_encoding:
if self._accessor.decode(utf16_encoding):
log.debug("encoding: guessed encoding: %r", utf16_encoding)
self.encoding = utf16_encoding
return
- # 11. Give UTF-8 a try.
+ # 10. Give UTF-8 a try.
norm_utf8_encoding = _norm_encoding("utf-8")
if self._accessor.decode(norm_utf8_encoding):
log.debug("UTF-8 encoding: %r", norm_utf8_encoding)
self.encoding = norm_utf8_encoding
return
+ # 11. Lang-specific fallback (e.g. XML -> utf-8, Python -> ascii, ...).
+ # Note: A potential problem here is that a fallback encoding here that
+ # is a pre-Unicode Single-Byte encoding (like iso8859-1) always "works"
+ # so the subsequent heuristics never get tried.
+ fallback_encoding = None
+ fallback_lang = None
+ if self.langinfo:
+ fallback_lang = self.langinfo.name
+ fallback_encoding = self.langinfo.conformant_attr("default_encoding")
+ if fallback_encoding:
+ if self._accessor.decode(fallback_encoding):
+ log.debug("encoding: fallback encoding for %s: %r",
+ fallback_lang, fallback_encoding)
+ self.encoding = fallback_encoding
+ return
+ else:
+ log.debug("encoding: %s fallback encoding (%r) was *wrong*",
+ fallback_lang, fallback_encoding)
+ self._encoding_bozo(
+ u"%s fallback encoding (%s) could not decode %s"
+ % (fallback_lang, fallback_encoding, self._accessor))
+
# 12. chardet (http://chardet.feedparser.org/)
# Note: I'm leary of using this b/c (a) it's a sizeable perf
# hit and (b) false positives -- for example, the first 8kB of
# /usr/bin/php on Mac OS X 10.4.10 is ISO-8859-2 with 44%
# confidence. :)
# Solution: (a) Only allow for content we know is not binary
# (from langinfo association); and (b) can be disabled via
# CHARDET_ENABLED class attribute.
if self.CHARDET_ENABLED and self.langinfo and self.langinfo.is_text:
try: