Re: second attempt at dstttr training

On Thu, Mar 5, 2009 at 2:27 PM, Thomas Michael Hagen
<[email protected]> wrote:
> i've been struggling with what you bring up as your big point as well,
> but this is how i understood the description of dstttr in the book (by
> bill).

See my pseudocode for the way to do [DS]STTTR. (The max rounds limit
is an extra, BTW. You will find the same sort of thing in mailtrainer,
for those who are interested in this sort of stuff (implementation vs
theory).)


> i have a different version of my classify, where the user is asked for
> input on every classification, but i was hoping that this one could
> run for a while before i got to that point, and somehow improve the
> accuracy.
>
> however, as you point out, it is a utopic illusion to think that the
> classifier is able to improve itself - at least by improving its own
> confidence in its weaker conclusions!
>
> i guess my next version will have to ask for user help whenever it is unsure.

Exactly!

That way, you will see (or I should say 'should') a fast decrease in
number of documents asked about.

When you test this sort of thing, it is VERY HANDY to have (or create)
a test set (a.k.a. corpus) which includes meta data, i.e. what
category each article should end up in. Can be as simple as an extra
file which has a list of articles as files and for each article the
category it ought to end up as.
That way, you can alter that pseudocode procedure a tiny bit (instead
of asking the human, you already pass the 'sollwert' to da_works() at
start as an extra parameter, so ask_human...() becomes a nil operation
and the test run can continue without any human intervention. That
means: start test run, drink coffee, peruse resulting report. No
attendance required anymore.




> i'll need to look more closely at your code before i can begin to
> incorporate; just one quick question: i'm assuming ist-wert means
> current value, and soll-wert means future value. is this about right?

Not exactly.

'sollwert' is German (and used in other parts of Europe too when it
comes to control engineering) for 'the value it SHOULD be' (literally:
MUST be. 'soll' is much stronger than 'muss') while 'istwert' means
'the value which it actually IS'. As such, 'sollwert' is the goal and
'istwert' is the actual state of affairs.

The basic control engineering 'control loop' takes a 'sollwert' and a
process-to-be-controlled (in our case: the article-to-category
classification) which takes an input (article) and produces an output
('istwert' / category).
This 'istwert' is sent through a 'feedback loop' and compared with the
desired 'sollwert'. This comparison results in an adjustment of the
process-to-be-controlled in order to make it come closer to the
desired goal ('sollwert'). Here, the adjustment process is the
classifier training operation.
It starts to get interesting when you consider the 'external
influences' which have an [adverse/uncontrolled] effect on the
process-to-be-controlled (error introduction ~ noise factors). Two
sections there: input noise and process noise. Input noise, in
software terms here, means data munching / preprocessing: it can help
to 'munch', i.e. preprocess the data before feeding it to the process
(in an attempt to reduce input noise levels). This can be anything
from data reduction to data augmentation.
'Process noise' is several components. One are bugs and/or design
flaws in your crm script or other parts of the software which you can
treat as white boxes. One other is inaccuracy / incapability of the
classifier used.[*]


For a picture:
  http://en.wikipedia.org/wiki/Control_theory
though I generally use something that looks more like this:
  http://en.wikibooks.org/w/index.php?title=File:System_5_Positions.png
which is nice & clean (at the cost of not showing all the various
process blocks)

As such, it also defines process scope.[**]



[*], [**] I've found that the hard part for some is mapping processes
onto this model as soon as you get outside the usual chem plant or
electronic rig, depending on where you first met control engineering.
Amazingly, control engineering doesn't get taught in CS at all. (At
least not over here and given my international experiences, not abroad
either. Instead, you'll have to make do with the derivatives in the
form of buzz, such as Agile, XP, etc. etc. and for the older boys,
waterfall. Which, incidentally, has a nice feedback loop, which
apparently nobody ever bothered with, once the exams were passed.
Humans are very much like pigs: there's only going forward. But I
digress.)



I'll bother you with the [*] and [**] related third (& fourth)
question later. Its answer(s) will have impact on your design and your
process/code flow, but best make this bit work like D/S-STTTR first
and get some results you like, and, with a little luck, you've already
found the answers for the questions to come, as well.





> ps: i've had complaints that my mails contain html-encoding. i think i
> have fixed that now. please let me know if i haven't.

Didn't notice it here; gmail is tolerant and the special, local
machine I also read mail on does the proper munching of such, so nil
gibberish, now or then.


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H