Re: Komodo 6 - find in files in utf8 without bom

I'd appreciate if we could move the discussion to http://bugs.activestate.com/show_bug.cgi?id=88698

The patch works, but I didn't see an obvious way to write a test for it.

In test_langinfo.py, trying langinfo.Database().langinfo_from_filename(new-file)
returns None

In Komodo the patch works fine.  Thanks, Trent

- Eric

On 11/6/2010 9:56 PM, Trent Mick wrote:
> Have a fix for it. The "problem" only happens for Perl files because
> Perl has a bone-headed default encoding: iso8859-1. :)
>
> Anyway, the problem is this:
>
> "textinfo.py"'s encoding detection heuristics will attempt a
> filetype's default encoding (in this case Perl's iso8859-1) before
> getting to the attempt at using UTF-8. For languages like XML (default
> UTF-8) and Python (default ASCII) the right thing happens with a file
> like the one I showed below. The problem is that a single-byte
> encoding like iso8859-1 always *appears* to work -- where "work" means
> that Python's `str.decode(ENCODING)` doesn't raise an exception.
>
> Solution is to move the "try the language's default encoding, if it
> has one" step in the encoding detection heuristics later in the
> process. A slightly better fix might be:
> - to only do this for those "single-byte encodings" that always "work"; and/or
> - move this heuristic step down *further* below the "chardet" step
> (see attached patch)
> But I don't think either of those "better" fixes are worth the bother.
>
> Patch attached.
>
> Karel,
> BTW, a quick hack fix is to remove the "default encoding" for Perl in
> your Komodo install something like this:
>
> {{{
> Index: langinfo_prog.py
> ===================================================================
> --- langinfo_prog.py    (revision 6654)
> +++ langinfo_prog.py    (working copy)
> @@ -69,21 +69,21 @@
>   class PerlLangInfo(LangInfo):
>       name = "Perl"
>       conforms_to_bases = ["Text"]
>       exts = ['.pl', '.pm', '.t']
>       magic_numbers = [
>           (0, "regex", re.compile(r'\A#!.*perl.*$', re.I | re.M)),
>       ]
>       filename_patterns = ["Construct", "Conscript"] # Cons
> make-replacement tool files
>
>       # http://search.cpan.org/~rgarcia/encoding-source-0.02/lib/encoding/source.pm
> -    default_encoding = "iso8859-1"
> +    #default_encoding = "iso8859-1"
>       # Perl>= 5.8.0
>       #   http://perldoc.perl.org/encoding.html
>       #   use encoding "<encoding-name>";
>       #   "Somewhat broken."
>       # Perl>= 5.9.5
>       #   http://search.cpan.org/~rgarcia/encoding-source-0.02/lib/encoding/source.pm
>       #   use encoding::source "<encoding-name>";
>       #   "This is like the encoding pragma, but done right."
>       encoding_decl_pattern = re.compile(
>           r"""use\s+encoding(?:::source)?\s+(['"])(?P<encoding>[\w-]+)\1""")
> }}}
>
>
>
>
>
> 2010/11/6 Trent Mick<[email protected]>:
>> "textinfo.py" (used by "Find in Files") is incorrectly guessing the
>> encoding of this file. As a result, "Find/Replace in Files" will mess
>> up for non-ASCII portions of the file.
>>
>> Eric,
>> Perhaps you could open a bug on textinfo.py?
>>
>> -----
>> $ cat findit.pl
>> I created a file containing the text "Vyváženě prožitý čas přeji",
>> set the encoding to utf-8, and could find the text
>> when the file was saved both with and without a BOM.
>> [20:56:47 trentm@banana:~/tmp/f]
>> $ file findit.pl
>> findit.pl: UTF-8 Unicode English text
>> [20:56:51 trentm@banana:~/tmp/f]
>> $ python ~/as/komodo/src/python-sitelib/textinfo.py findit.pl
>> findit.pl: Perl, iso8859-1
>> ----
>>
>>
>> Trent
>>
>> --
>> Trent Mick
>>
>
>
>
>
>
> _______________________________________________
> Komodo-discuss mailing list
> [email protected]
> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
> Other options: http://listserv.ActiveState.com/mailman/listinfo/Komodo-discuss
_______________________________________________
Komodo-discuss mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Other options: http://listserv.ActiveState.com/mailman/listinfo/Komodo-discuss