Semantic Interpretation of Block Structure

One of the characteristics of Spam is that the Spammers are trying to
hide the semantic structure of their content from classifiers, but that 
isn't true of other classification applications. Generally speaking,
the beginning and end of block structures are significant queues to
semantic content that are missed in tokenization techniques that treat
all white space as equal.

I'm looking for tokenization strategies to appropriately weight semantic
mark up, specifically whether anyone has already experimented with the
effect of using tokens for paragraph and other block elements.

Stictly speaking, a block delineator should probably not have the same
weight as a word, but should instead reduce the relative weight that
two words in different blocks have a combination... which for Markovian
classifiers is probably functionally equivalent.

Has anyone experimented with this already?

Chris

------------------------------------------------------------------------------
Stay on top of everything new and different, both inside and 
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. 
Use priority code J9JMT32. http://p.sf.net/sfu/p

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iQEcBAEBAgAGBQJJ62G2AAoJEASgqNsqZfCHoOoIALfkdmsPb+Ekhxx8PEuS/TSa
fyK1RwIjU5kMBQuQ9JYuHI7HcAMIKG9lGuVv7WgsNs0jNxl8cmAZvUuOvlMhJRhp
DXYeL4XHRTEYkoJ9+ewUmABjDgScdOJqaURiiMfdVsZiZ81zcRkHhvsjV1vZCuhC
3XjKWTgZibNNAWQyZlNGszI3qSPihyms6jiV9tE+lW9FUOuPLLnpBXKsmbMlOqOY
RByxGFQ2Tavjfrs/oUX7Ts0GKehy5skhtjStr2yH8hiqiO6saNZ2D5kUPYyUhhX7
kNs99CMTUQJKHLsx/BjUnVpt+Y/N28w4WxV8Mnu/eQIWvTGp4UMeUqlprcxGxhY=
=NVFP
-----END PGP SIGNATURE-----