Re: filtering out non-Japanese

On Wed, Dec 15, 2004 at 02:49:46PM +0000, John Delacour wrote:
: At 12:39 pm +0100 15/12/04, Marco Baroni wrote:
: 
: >where can I find the hexadecimal hiragana, katakana and kanj ranges?

$ cat uni
#!/usr/bin/perl -C
 
binmode STDOUT, ":utf8";
$pat = shift;
if (ord $pat > 256) {
    $pat = sprintf("%04x", ord $pat);
}
elsif (ord $pat > 128) {        # arg in sneaky UTF-8
    $pat = sprintf("%04x", unpack("U0U",$pat));
}
 
@names = split /^/, do 'unicore/Name.pl';
for (@names) {
    if (/$pat/io) {
        $hex = hex($_);
        print chr($hex),"\t",$_;
    }
}

$ ./uni hiragana
ぁ	3041		HIRAGANA LETTER SMALL A
あ	3042		HIRAGANA LETTER A
ぃ	3043		HIRAGANA LETTER SMALL I
い	3044		HIRAGANA LETTER I
ぅ	3045		HIRAGANA LETTER SMALL U
う	3046		HIRAGANA LETTER U
ぇ	3047		HIRAGANA LETTER SMALL E
え	3048		HIRAGANA LETTER E
ぉ	3049		HIRAGANA LETTER SMALL O
お	304A		HIRAGANA LETTER O
か	304B		HIRAGANA LETTER KA
が	304C		HIRAGANA LETTER GA
き	304D		HIRAGANA LETTER KI
ぎ	304E		HIRAGANA LETTER GI
く	304F		HIRAGANA LETTER KU
ぐ	3050		HIRAGANA LETTER GU
け	3051		HIRAGANA LETTER KE
げ	3052		HIRAGANA LETTER GE
こ	3053		HIRAGANA LETTER KO
ご	3054		HIRAGANA LETTER GO
さ	3055		HIRAGANA LETTER SA
ざ	3056		HIRAGANA LETTER ZA
し	3057		HIRAGANA LETTER SI
じ	3058		HIRAGANA LETTER ZI
す	3059		HIRAGANA LETTER SU
ず	305A		HIRAGANA LETTER ZU
せ	305B		HIRAGANA LETTER SE
ぜ	305C		HIRAGANA LETTER ZE
そ	305D		HIRAGANA LETTER SO
ぞ	305E		HIRAGANA LETTER ZO
た	305F		HIRAGANA LETTER TA
だ	3060		HIRAGANA LETTER DA
ち	3061		HIRAGANA LETTER TI
ぢ	3062		HIRAGANA LETTER DI
っ	3063		HIRAGANA LETTER SMALL TU
つ	3064		HIRAGANA LETTER TU
づ	3065		HIRAGANA LETTER DU
て	3066		HIRAGANA LETTER TE
で	3067		HIRAGANA LETTER DE
と	3068		HIRAGANA LETTER TO
ど	3069		HIRAGANA LETTER DO
な	306A		HIRAGANA LETTER NA
に	306B		HIRAGANA LETTER NI
ぬ	306C		HIRAGANA LETTER NU
ね	306D		HIRAGANA LETTER NE
の	306E		HIRAGANA LETTER NO
は	306F		HIRAGANA LETTER HA
ば	3070		HIRAGANA LETTER BA
ぱ	3071		HIRAGANA LETTER PA
ひ	3072		HIRAGANA LETTER HI
び	3073		HIRAGANA LETTER BI
ぴ	3074		HIRAGANA LETTER PI
ふ	3075		HIRAGANA LETTER HU
ぶ	3076		HIRAGANA LETTER BU
ぷ	3077		HIRAGANA LETTER PU
へ	3078		HIRAGANA LETTER HE
べ	3079		HIRAGANA LETTER BE
ぺ	307A		HIRAGANA LETTER PE
ほ	307B		HIRAGANA LETTER HO
ぼ	307C		HIRAGANA LETTER BO
ぽ	307D		HIRAGANA LETTER PO
ま	307E		HIRAGANA LETTER MA
み	307F		HIRAGANA LETTER MI
む	3080		HIRAGANA LETTER MU
め	3081		HIRAGANA LETTER ME
も	3082		HIRAGANA LETTER MO
ゃ	3083		HIRAGANA LETTER SMALL YA
や	3084		HIRAGANA LETTER YA
ゅ	3085		HIRAGANA LETTER SMALL YU
ゆ	3086		HIRAGANA LETTER YU
ょ	3087		HIRAGANA LETTER SMALL YO
よ	3088		HIRAGANA LETTER YO
ら	3089		HIRAGANA LETTER RA
り	308A		HIRAGANA LETTER RI
る	308B		HIRAGANA LETTER RU
れ	308C		HIRAGANA LETTER RE
ろ	308D		HIRAGANA LETTER RO
ゎ	308E		HIRAGANA LETTER SMALL WA
わ	308F		HIRAGANA LETTER WA
ゐ	3090		HIRAGANA LETTER WI
ゑ	3091		HIRAGANA LETTER WE
を	3092		HIRAGANA LETTER WO
ん	3093		HIRAGANA LETTER N
ゔ	3094		HIRAGANA LETTER VU
ゕ	3095		HIRAGANA LETTER SMALL KA
ゖ	3096		HIRAGANA LETTER SMALL KE
゙	3099		COMBINING KATAKANA-HIRAGANA VOICED SOUND MARK
゚	309A		COMBINING KATAKANA-HIRAGANA SEMI-VOICED SOUND MARK
゛	309B		KATAKANA-HIRAGANA VOICED SOUND MARK
゜	309C		KATAKANA-HIRAGANA SEMI-VOICED SOUND MARK
ゝ	309D		HIRAGANA ITERATION MARK
ゞ	309E		HIRAGANA VOICED ITERATION MARK
ゟ	309F		HIRAGANA DIGRAPH YORI
゠	30A0		KATAKANA-HIRAGANA DOUBLE HYPHEN
ー	30FC		KATAKANA-HIRAGANA PROLONGED SOUND MARK
ｰ	FF70		HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK