Re: Questions for periodicPruning

Hi Eibe,

Thanks for your response although it's not a fast one 😀


Hi Andria (aka Edward aka Valerio aka Philipp?),
>

Although I'm not sure why you're saying that, but Andria is me.


> 1. The idea is to prevent StringToWordVector from running out of memory by
> periodically discarding words (really, tokens), that don't meet the minimum
> frequency threshold. There is no reason to use periodic pruning if you
> don't run out of memory.
>
> 2. Whatever number is small enough to prevent a crash due to an
> out-of-memory error.
>
> 3. Periodic pruning makes use of the minimum frequency parameter. The
> latter interacts with wordsToKeep when the *final* dictionary is
> constructed: if the number of tokens in the dictionary obtained after
> processing all the training text is smaller than wordsToKeep, the final
> dictionary will be pruned based on the minimum frequency threshold;
> otherwise, it will be pruned based on wordsToKeep.
>
> Note that wordsToKeep=1000, for example, may not give you exactly 1,000
> dictionary entries because StringToWordVector does not break ties: it will
> keep all tokens that are at least as frequent as the 1,000th most frequent
> token.
>
> The relevant bit of code for periodic pruning is
>
> /**
>  * Prunes the dictionary of low frequency terms
>  */
> protected void pruneDictionary() {
>   if (m_periodicPruneRate > 0 && m_count % m_periodicPruneRate == 0) {
>     for (Map<String, int[]> m_dictsPerClas : m_dictsPerClass) {
>       Iterator<Map.Entry<String, int[]>> entries =
>         m_dictsPerClas.entrySet().iterator();
>       while (entries.hasNext()) {
>         Map.Entry<String, int[]> entry = entries.next();
>         if (entry.getValue()[0] < m_minFrequency) {
>           entries.remove();
>         }
>       }
>     }
>   }
> }
>

Wonderful! It's very clear now. This wasn't clear enough in the filter's
description.


> Finally, I respect your desire for anonymity, and appreciate your interest
> in WEKA and ADAMS, but it would be more motivating for us to answer your
> questions if we knew who you are.
>
It's nice to get to know the people who use our software. Just a thought.
> Also, I don't think using multiple aliases will generally help you get a
> quicker response. And repeatedly posting the same question won't help
> either.
>

Finally, posting a question that found to be interesting by another member is
only to get an answer just like anyone here. Speedy answers is not my goal
(you can see that I'm waiting your generosity to answer my question for a
few days and did not complain 😀). Eibe, I do appreciate your effort in
answering people's questions. It would be nice as well as professional to
help people regardless to their names, religions, etc. which I believe name
or even a person means nothing to you--do you why? because several times
people (aka Weka users) get insulted in this list but you were quite and
you did not talk like now. Hence, my advice is that it would be nice to
focus on supporting Weka users and getting every one in this list respected
rather than getting his/her ID-related information as an excuse for you to
have a motivation to reply them in this list.

Cheers,
Andria




> Cheers,
> Eibe
>
> On Wed, Jun 16, 2021 at 1:25 PM Andria Lan <[email protected]> wrote:
>
>> Hi all,
>>
>> In StringToWordVector, I don't understand the role behind periodicPruning
>> parameter although it is stated that the filter is used to specify the rate
>> (x% of the input dataset) at which to periodically prune the dictionary.
>>
>> In sum, here are my questions:
>> 1- What is the idea behind the periodicPruning?
>> 2- How to select the number to be used for the  periodicPruning?
>> 3- What is the relation between "wordsToKeep" parameter and
>> periodicPruning parameter?
>>
>> Thank you.
>>
>> Andria
>> _______________________________________________
>> Wekalist mailing list -- [email protected]
>> Send posts to [email protected]
>> To unsubscribe send an email to [email protected]
>> To subscribe, unsubscribe, etc., visit
>> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
>> List etiquette:
>> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>>
> _______________________________________________
> Wekalist mailing list -- [email protected]
> Send posts to [email protected]
> To unsubscribe send an email to [email protected]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette:
> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>

_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html