Re: Alphabets Benchmarks - How many ways to unaccent a text string? Turn AÄÁaäá into AAAaaa. And the winner is...

Hello,

  Great thanks. Today I learned that String#gsub can take a Hash as
its second argument. I added your unaccent function.

   About tr - that's great too and I guess that's as fast as you can
get - but unaccent will not work with ligatures e.g. 'æ'=>'ae', 'ß' =>
'ss' or german umlaut transliteration 'ä' => 'ae', 'ö' => 'oe' etc.

    Some more new examples include - to quote from the updated readme [1]:

Samuel Williams writes in with one more optimization.
Why not replace the `NON_ALPHA_CHAR_REGEX`, that is, `/[^A-Za-z0-9 ]/`
with a regex matching only known accented chars?

``` ruby
UNACCENT_REGEX = Regexp.union( UNACCENT.keys )
def unaccent_gsub_v3b( text, mapping=UNACCENT, regex=UNACCENT_REGEX )
  text.gsub( regex, mapping)
end
```


Hold on. Let's add some more optimizations to the humble `each_char`
version too.
For all 7-bit (less than 0x7F) unicode latin basic (also known as ascii)
char(acter)s no mapping (ever) needed. Let's try:

``` ruby
def unaccent_each_char_v2_7bit( text, mapping )
  buf = String.new
  text.each_char do |ch|
    buf <<   if ch.ord < 0x7F
               ch
             else
               mapping[ch] || ch
             end
  end
  buf
end
```

Maybe the mapping lookup using an array index by an integer number
is faster than hash mapping lookup by single-character string?
Let's try:

``` ruby
UNACCENT_FASTER = UNACCENT.reduce( [] ) do |ary,(ch,value)|
  ary[ ch.ord ] = value
  ary
end

def unaccent_each_char_v2_7bit_faster( text, mapping_faster=UNACCENT_FASTER )
  buf = String.new
  text.each_char do |ch|
    buf <<  if ch.ord < 0x7F
               ch
            else
               mapping_faster[ ch.ord ] || ch
            end
  end
  buf
end
```

     Voila. And the winner is...      Can you find a faster way? Show us.

   Happy data (and text) wrangling with ruby. Cheers. Prost.

[1] https://github.com/sportdb/sport.db/tree/master/alphabets/benchmark

Unsubscribe: <mailto:[email protected]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>