Re: second attempt at dstttr training

On Thu, Mar 5, 2009 at 10:28 AM, Thomas Michael Hagen
<[email protected]> wrote:
> here's my second attempt, after valuable input from ger:
>
> comments and harsh criticism is very welcome :)

Had to stare at it for a little while... anyway, harsh criticism is my forte ;-)

What I see happening in your classify code is this, written in
'Hobbelt'[*] pseudocode:

------------------------------------
function classify(article)
{
	crm114.classify(article, CFC_set) -> [pR, class_conc]
	if (failed = no)
	{
		# CRM114 selected a cfc file; the 'winner'
		# lands in :class_conc:
		if (pR > 30)
		{
			print "Classify succeeded as $class_conc"
			return
		}
	
		# pR wasn't high enough; accumulating statistics:
		conc_file <- class_conc ".cfc"
		crm114.learn(article, conc_file)
		# GHO: are you absolutely positively sure you want that?
	
		print "Classify succeeded, but threshold low. ..."
	
		# reclassifying to check improvement:
		
		crm114.classify(article, CFC_set) -> [pR2, ---]
		imp <- pR2 - pR
		if (pR > 10 && imp > 3)
		{
			print "Improvement high enough, ..."
			return
		}
		
		# imp wasn't high enough, and new threshold wasn't met
	
		refute(article, conc_file)
		return
	}
	else
	{
		# classify failed (category is unc or out)
		learn (article)
		if (failed = yes)
		{
			# did learn fail?:
			interaction(...)
			return
		}
		else
		{
			print "Learn succeeded ... after classify FAILED ..."
			return
		}
	}
}
------------------------------------

I find the pseudocode a little easier to grok than crm114 script, but
that's probably just me.

Anyway, here's the harsh criticism 'n all:

>From a code flow perspective, nothing to nag about.

>From a functional flow perspective, the big one remains: are you
absolutely positively 100% sure you want to crm114.learn() an article
into a category (CFC) based on an automated classification? NOT
especially when that classification didn't make it in the match
olympics already, i.e. is definitely on the shady side.


Okay, let me chop that one up for you:

1) do you want to train an article into the category GUESTIMATED by an
automaton, when that automaton already indicated, according to your
own standards, that 'it isn't really sure about this'?
Try to rephrase this with human actors (roleplay), where the librarian
isn't exactly sure where to put that item he's got. Exercise question
to finish this roleplay: how do you rate your chances at retrieving
articles managed/archived by a librarian who 'isn't exactly sure about
it'? Hmmmm.

2) do you want to train an article into a category GUESTIMATED by an
automaton, period?
A classifier can be viewed as a retard ^H^H^H^H^H^H^H mentally
challenged human down on his luck, which you've put in charge of the
library. We've done our MBA, read our Dilbert so we enjoy the A/P
skyrocketing towards black numbers again, thanks to this initiative,
while we tolerate the slight reduction in performance. That is: the
automaton can do retrieval for us fine. But we ARE aware it WILL make
[a few] mistakes. Which is perfectly acceptable to us.
Now ask ourselves the #2 here: do we want such a process/individual to
steer our archival process as well? One mistake in archival will
produce a PERMANENT error as such a misplaced article will never be
found. Until we invest a lot of $$$ to sift through the entire library
(reorganization).


Of course, #2 is disputable as some members may argue that the
classifiers are _smarter_ than the human type of same. Unfortunately,
I don't attend that church. The only thing in my book that makes
automated classifiers perform better than humans is because:
a) the don't fall asleep / don't strike for higher pay
b) at least attempt to be long term consistent (read: are unemotional)


Given 1) + 2) I'd rather expect a flow like this:

------------------------------------------
function da_works(article)
{
	// ist-wert
	classify(article) -> [sure, pR, category]
	if (sure)
	{
		print "assigned category:", category
		return category
	}
	# else: not ok
	go_ask_human_where_this_belongs(article) -> soll_wert

	# training SSTTTR/DSTTTR now:
	rounds = 5
	# ^^^ example: upper bound to keep things sane in the
	# bayes box
	do
	{
		crm114.train(article, soll_wert.as_category)

		# check 'thick' threshold:
		classify(article) -> [sure, pR, category]
		
		# in case of threshold identical to regular
		# classify margin:
		--> if (sure) exit loop

		# otherwise, when you want to use a different
		# 'thick threshold', e.g. like mailtrainer,
		# which has/had the regular threshold at
		# pR = +10/-5.0 (assym!) while the 'thick'
		# threshold is +/- 5.0:
		if (pR > 5.0) exit loop
		
		# and another round!
		decrement rounds
	} while rounds > 0

	if (sure)
	{
		print "trained, and we're sure now"
		return category
	}
	else
	{
		print "trained, yet the automaton still didn't 'get' it"
		return soll_wert.as_category
	}
}


function classify(article)
{
	crm114.classify(article, cfc_set) -> [pR, category]
	if (failed)
	{
		return NOT SURE, 0.0, unknown_category
	}
	else if (pR > 30)
	{
		return SURE, pR, category
	}
	else
	{
		return NOT SURE, pR, category
	}
}
------------------------------------------


Of course, didn't get to see your own :learn: and other bits, but that
doesn't really matter: the Q1 and Q2 stand and I only wrote the whole
process (da_works()) as pseudocode so you can see what I'd come up
with. In any language / software process.

The key part is that, as soon as the automaton starts to make unsure
noises, I defer to a human. No training is done, unless the human
tells the machine what to do.
It's the 'I don't know where this goes' librarian roleplay again:
there's the human [supervisor] to tell the librarian where to put that
bit he just got and the librarian is not permitted to just put it
anywhere; if this dilemma would block the flow, than countermeasures
(SOPs aka Standard Operating Procedures) are in place, I'm sure, so
the librarian can go on doing other work he IS sure about, until this
item is resolved by his super.

Basic library management out in a classifier/software environment. So
we all can find our articles and books in the future. ;-)



In closing: I am sure you've designed your process like this and made
those important design decisions before you started working on that
crm114 script. It's just that sometimes design decisions can get
obscured during the implementation phase. Hence me asking them again
here.
And as that 'sollwert' (and the implicit classified 'istwert') may
have hinted to you already, this is control engineering (it's just
that software engineers don't realize it). Which means there's still
one important question unasked.

Nice work so far, keep it up!



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H