Re: second attempt at dstttr training

   From: Thomas Michael Hagen <[email protected]>

   here's my second attempt, after valuable input from ger:

   comments and harsh criticism is very welcome :)

It can be risky to do completely automated training (that is, training
an article into the set it fit closest to, which is what I think
you're doing, as you set :class_conc: as a match of part of :stats:.)

This is akin to clustering and one very real danger is that one or two
clusters starts growing uncontrollably and starts to take over
everything.

So, I would be careful with that.  It might work out for incremental
training, but please be careful; it may start diverging and go
unstable.

I also see that you do DSTTTR auto-refute out of all other classes.
That's pretty intense training, but if it works, great.  The one
rule of thumb I have learned is that there is no rule of thumb here.

   - Bill Yerazunis


   :classify: (:article:)
   {
       {
       isolate (:stats:)
       classify <osbf microgroom> [:article:] \
	   (bus.cfc cli.cfc con.cfc cul.cfc dom.cfc \
	   eco.cfc edu.cfc ene.cfc ent.cfc env.cfc \
	   for.cfc hea.cfc loc.cfc pol.cfc sci.cfc \
	   spo.cfc tec.cfc tra.cfc wor.cfc | unc.cfc out.cfc) (:stats:)
       match (:: :class-conc:) [:stats:] /(...)\.cfc/
       { #classify succeeded, checking pR:
	   match (:: :pR:) [:stats:] /Best match.*: (.*)  \nTotal/
	   output /pR::*:pR:. /
	   eval /:@::*:pR: > 30:/ #play around with this number
	   output /Classify succeeded as ':*:class-conc:'\n/
	   return
       } #pR wasn't high enough; accumulating statistics:
       isolate (:conc-file:) /:*:class-conc:.cfc/
       learn <osbf microgroom> (:*:conc-file:)  [:article:]
       output /Classify succeeded, but threshold low. Piece trained INTO
   ':*:class-conc:'\n/
       #reclassifying to check improvement:
       isolate (:stats2:)
       classify <osbf microgroom> [:article:]    \
	   (bus.cfc cli.cfc con.cfc cul.cfc dom.cfc \
	   eco.cfc edu.cfc ene.cfc ent.cfc env.cfc \
	   for.cfc hea.cfc loc.cfc pol.cfc sci.cfc \
	   spo.cfc tec.cfc tra.cfc wor.cfc | unc.cfc out.cfc) (:stats2:)
       match (:: :pR2:) [:stats2:] /Best match.*: (.*)  \nTotal/
       isolate (:improvement:)
       eval (:improvement:) /:@:(:*:pR2: - :*:pR:):/
       output /imp::*:improvement:\n/
       {
	   eval /:@::*:improvement: > 3:/
	   eval /:@::*:pR2: > 10:/ #play around with this number too
	   output /Improvement high enough, and new threshold met; NO further
   action taken\n/
	   return
       } # imp wasn't high enough, and new threshold wasn't met
       isolate (:refute-args:) /:*:conc-file::*:article:/
       call /:refute:/ [:*:refute-args:] #this refutes the article from all the
   other categories
       return
       } # classify failed (category is unc or out)
       # my ad-hoc learning function (simply looks at the url to find
   keywords):
       call /:learn:/ [:*:article:]
       { # did learn fail?:
       match [:learn-conc:] /unc/
       match (:: :class-conc:) [:stats:] /(...)\.cfc/ #yes, has to be called
   again here
       isolate (:interaction:) /:*:class-conc::*:article::*:stats:/
       call /:interact:/ [:*:interaction:] #ask a human for the right category
       return
       } # learn succeeded
       output /Learn succeeded as ':*:learn-conc:' after classify FAILED as
   ':*:class-conc:'\n/
       return
   }

   --000e0cd248049b082c04645bcab5
   Content-Type: text/html; charset=ISO-8859-1
   Content-Transfer-Encoding: quoted-printable

   here&#39;s my second attempt, after valuable input from ger:<br><br>comment=
   s and harsh criticism is very welcome :)<br><br>:classify: (:article:)<br>{=
    <br>=A0=A0=A0 {<br>=A0=A0=A0 isolate (:stats:)<br>=A0=A0=A0 classify &lt;o=
   sbf microgroom&gt; [:article:] \<br>
   =A0=A0=A0 =A0=A0=A0 (bus.cfc cli.cfc con.cfc cul.cfc dom.cfc \<br>=A0=A0=A0=
    =A0=A0=A0 eco.cfc edu.cfc ene.cfc ent.cfc env.cfc \<br>=A0=A0=A0 =A0=A0=A0=
    for.cfc hea.cfc loc.cfc pol.cfc sci.cfc \<br>=A0=A0=A0 =A0=A0=A0 spo.cfc t=
   ec.cfc tra.cfc wor.cfc | unc.cfc out.cfc) (:stats:)<br>
   =A0=A0=A0 match (:: :class-conc:) [:stats:] /(...)\.cfc/<br>=A0=A0=A0 { #cl=
   assify succeeded, checking pR:<br>=A0=A0=A0 =A0=A0=A0 match (:: :pR:) [:sta=
   ts:] /Best match.*: (.*)=A0 \nTotal/<br>=A0=A0=A0 =A0=A0=A0 output /pR::*:p=
   R:. /<br>=A0=A0=A0 =A0=A0=A0 eval /:@::*:pR: &gt; 30:/ #play around with th=
   is number<br>
   =A0=A0=A0 =A0=A0=A0 output /Classify succeeded as &#39;:*:class-conc:&#39;\=
   n/<br>=A0=A0=A0 =A0=A0=A0 return<br>=A0=A0=A0 } #pR wasn&#39;t high enough;=
    accumulating statistics:<br>=A0=A0=A0 isolate (:conc-file:) /:*:class-conc=
   :.cfc/<br>=A0=A0=A0 learn &lt;osbf microgroom&gt; (:*:conc-file:)=A0 [:arti=
   cle:]<br>
   =A0=A0=A0 output /Classify succeeded, but threshold low. Piece trained INTO=
    &#39;:*:class-conc:&#39;\n/<br>=A0=A0=A0 #reclassifying to check improveme=
   nt:<br>=A0=A0=A0 isolate (:stats2:)<br>=A0=A0=A0 classify &lt;osbf microgro=
   om&gt; [:article:]=A0=A0=A0 \<br>
   =A0=A0=A0 =A0=A0=A0 (bus.cfc cli.cfc con.cfc cul.cfc dom.cfc \<br>=A0=A0=A0=
    =A0=A0=A0 eco.cfc edu.cfc ene.cfc ent.cfc env.cfc \<br>=A0=A0=A0 =A0=A0=A0=
    for.cfc hea.cfc loc.cfc pol.cfc sci.cfc \<br>=A0=A0=A0 =A0=A0=A0 spo.cfc t=
   ec.cfc tra.cfc wor.cfc | unc.cfc out.cfc) (:stats2:)<br>
   =A0=A0=A0 match (:: :pR2:) [:stats2:] /Best match.*: (.*)=A0 \nTotal/<br>=
   =A0=A0=A0 isolate (:improvement:)<br>=A0=A0=A0 eval (:improvement:) /:@:(:*=
   :pR2: - :*:pR:):/<br>=A0=A0=A0 output /imp::*:improvement:\n/<br>=A0=A0=A0 =
   {<br>=A0=A0=A0 =A0=A0=A0 eval /:@::*:improvement: &gt; 3:/<br>
   =A0=A0=A0 =A0=A0=A0 eval /:@::*:pR2: &gt; 10:/ #play around with this numbe=
   r too<br>=A0=A0=A0 =A0=A0=A0 output /Improvement high enough, and new thres=
   hold met; NO further action taken\n/<br>=A0=A0=A0 =A0=A0=A0 return<br>=A0=
   =A0=A0 } # imp wasn&#39;t high enough, and new threshold wasn&#39;t met<br>
   =A0=A0=A0 isolate (:refute-args:) /:*:conc-file::*:article:/<br>=A0=A0=A0 c=
   all /:refute:/ [:*:refute-args:] #this refutes the article from all the oth=
   er categories<br>=A0=A0=A0 return <br>=A0=A0=A0 } # classify failed (catego=
   ry is unc or out)<br>
   =A0=A0=A0 # my ad-hoc learning function (simply looks at the url to find ke=
   ywords):<br>=A0=A0=A0 call /:learn:/ [:*:article:]<br>=A0=A0=A0 { # did lea=
   rn fail?:<br>=A0=A0=A0 match [:learn-conc:] /unc/<br>=A0=A0=A0 match (:: :c=
   lass-conc:) [:stats:] /(...)\.cfc/ #yes, has to be called again here<br>
   =A0=A0=A0 isolate (:interaction:) /:*:class-conc::*:article::*:stats:/<br>=
   =A0=A0=A0 call /:interact:/ [:*:interaction:] #ask a human for the right ca=
   tegory<br>=A0=A0=A0 return<br>=A0=A0=A0 } # learn succeeded<br>=A0=A0=A0 ou=
   tput /Learn succeeded as &#39;:*:learn-conc:&#39; after classify FAILED as =
   &#39;:*:class-conc:&#39;\n/<br>
   =A0=A0=A0 return<br>}<br><br>

   --000e0cd248049b082c04645bcab5--


   --===============9159009357422728126==
   Content-Type: text/plain; charset="us-ascii"
   MIME-Version: 1.0
   Content-Transfer-Encoding: 7bit
   Content-Disposition: inline

   ------------------------------------------------------------------------------
   Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
   -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
   -Strategies to boost innovation and cut costs with open source participation
   -Receive a $600 discount off the registration fee with the source code: SFAD
   http://p.sf.net/sfu/XcvMzF8H
   --===============9159009357422728126==
   Content-Type: text/plain; charset="us-ascii"
   MIME-Version: 1.0
   Content-Transfer-Encoding: 7bit
   Content-Disposition: inline

   _______________________________________________
   Crm114-general mailing list
   [email protected]
   https://lists.sourceforge.net/lists/listinfo/crm114-general

   --===============9159009357422728126==--


------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H