Re: Questions for periodicPruning

Hi Andria (aka Edward aka Valerio aka Philipp?),

1. The idea is to prevent StringToWordVector from running out of memory by
periodically discarding words (really, tokens), that don't meet the minimum
frequency threshold. There is no reason to use periodic pruning if you
don't run out of memory.

2. Whatever number is small enough to prevent a crash due to an
out-of-memory error.

3. Periodic pruning makes use of the minimum frequency parameter. The
latter interacts with wordsToKeep when the *final* dictionary is
constructed: if the number of tokens in the dictionary obtained after
processing all the training text is smaller than wordsToKeep, the final
dictionary will be pruned based on the minimum frequency threshold;
otherwise, it will be pruned based on wordsToKeep.

Note that wordsToKeep=1000, for example, may not give you exactly 1,000
dictionary entries because StringToWordVector does not break ties: it will
keep all tokens that are at least as frequent as the 1,000th most frequent
token.

The relevant bit of code for periodic pruning is

/**
 * Prunes the dictionary of low frequency terms
 */
protected void pruneDictionary() {
  if (m_periodicPruneRate > 0 && m_count % m_periodicPruneRate == 0) {
    for (Map<String, int[]> m_dictsPerClas : m_dictsPerClass) {
      Iterator<Map.Entry<String, int[]>> entries =
        m_dictsPerClas.entrySet().iterator();
      while (entries.hasNext()) {
        Map.Entry<String, int[]> entry = entries.next();
        if (entry.getValue()[0] < m_minFrequency) {
          entries.remove();
        }
      }
    }
  }
}

Finally, I respect your desire for anonymity, and appreciate your interest
in WEKA and ADAMS, but it would be more motivating for us to answer your
questions if we knew who you are. It's nice to get to know the people who
use our software. Just a thought. Also, I don't think using multiple
aliases will generally help you get a quicker response. And repeatedly
posting the same question won't help either.

Cheers,
Eibe

On Wed, Jun 16, 2021 at 1:25 PM Andria Lan <[email protected]> wrote:

> Hi all,
>
> In StringToWordVector, I don't understand the role behind periodicPruning
> parameter although it is stated that the filter is used to specify the rate
> (x% of the input dataset) at which to periodically prune the dictionary.
>
> In sum, here are my questions:
> 1- What is the idea behind the periodicPruning?
> 2- How to select the number to be used for the  periodicPruning?
> 3- What is the relation between "wordsToKeep" parameter and
> periodicPruning parameter?
>
> Thank you.
>
> Andria
> _______________________________________________
> Wekalist mailing list -- [email protected]
> Send posts to [email protected]
> To unsubscribe send an email to [email protected]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette:
> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>

_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html