filtering out non-Japanese

[email protected] (Marco Baroni)
Newsgroups perl.unicode
Message-ID <[email protected]>
Dear all,

I have a long text ostensibly in utf-8, and I would like to get rid of 
all the lines that contain anything BUT kanji, katakana or hiragana 
(thus, throwing away Latin, but also digits, punctuation, etc.)

In short, I would like to do something like:

perl -ne 'if (/[^\p{Hiragana}\p{Katakana}\p{Kanji}]/){print}' 
webcorpus.tok > webcorpus.clean.tok

Is is possible to do something like that?

Thanks a lot!

Marco
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.