Re: Bug in wekaPython

Dear Mark,

actually, this is possible and worked for every package I tried so far. 
You just have to download the Jar and include it (and the dependencies), 
similar to how you can just use Weka itself as a library.

With wekaPython this is more tricky, due to the two python scripts 
pyCheck.py and pyServer.py.

But I may have misread the code. It seems like the py-path parameters 
are just added to the system path prior to calling the scripts. I 
thought that this parameter could also be used to provide the location 
of the scripts. But that does not seem to be the case, so the solution 
seems to be to just copy them to 
~/wekafiles/packages/wekaPython/resources/py.

A small help for me for finding this problem (or other config-issues 
with this package) could be to extend the error message of the 
IOException in checkLocalSocketAndCreateShutdownHook of the 
PythonServer. Currently, this only states that the server could not 
start. Since this is only used after starting the server, and the most 
likely mistake is that a) python is not found or b) the scripts are not 
found, the message could possibly be extended as follows:

"Was unable to start python server. Check if python is available on the 
path and if the required resources are available within the weka 
packages directory."

Also, we found a memory leak. We conformed this for the 
ScikitLearnClassifier, but its probably also in the other classes. The 
parameter m_removeModelFromPyPostTraining is fixed, depending on the 
classifier. For the RandomForestClassifier, this is set to false. We 
train many random forests, all of which are kept in the running python 
interpreter with no (non-hacky-reflection) way to clean this. Currently, 
they would only be cleaned when the PythonServer is shut down, which 
only happens when the JVM exists. This drains our memory in long-running 
experiments.

A possible solution would be to refactor the cleanUp code into a public 
method, that is called from buildClassifier and distributionForInstance, 
where it is currently duplicated. This method could be called by users 
for clean up, when the classifier is not required anymore.

Best,
Steffen

Am 09/06/2021 um 03:18 schrieb Mark Hall:
> Hi Steffen,
>
> I’m not too sure I understand the problem. The package (and the 
> server) cannot be used unless the package is installed by Weka’s 
> package manager.
>
> Cheers,
> Mark.
> On 9 Jun 2021, 4:04 AM +1200, Steffen Herbold 
> <[email protected]>, wrote:
>> Dear Devs,
>>
>> I think I found a bug in wekaPython. It seems like the py-path parameter
>> is not always used as it should by the PythonServer.
>>
>> I encountered the problem in a batch system, in which I did not have the
>> GUI and, therefore, did not use the package manager. Here is the
>> problematic code:
>>
>>   private void launchServer(boolean startPython) throws IOException {
>>     Thread acceptThread = startServerSocket();
>>     int localPort = m_serverSocket.getLocalPort();
>>
>>     if (startPython) {
>>       String serverScript =
>> WekaPackageManager.PACKAGES_DIR.getAbsolutePath()
>>         + File.separator + "wekaPython" + File.separator + "resources"
>>         + File.separator + "py" + File.separator + "pyServer.py";
>>       ProcessBuilder processBuilder = new ProcessBuilder(m_pythonCommand,
>>         serverScript, "" + localPort, m_debug ? "debug" : "");
>>       m_serverProcess = processBuilder.start();
>>     }
>>     try {
>>       acceptThread.join();
>>     } catch (InterruptedException e) {
>>     }
>>
>>     checkLocalSocketAndCreateShutdownHook();
>>   }
>>
>> The serverScript String ignores the py-path, which results in failures
>> because the server cannot be started if there is no packages folder with
>> the required ~/wekaPython/resources/py/* files.
>>
>> Best,
>> Steffen
>>
>> --
>> PD Dr. Steffen Herbold
>> Institute of Computer Science, University of Goettingen
>> Goldschmidtstraße 7, 37077 Göttingen, Germany
>> mailto. [email protected]
>> tel. +49 551 39-172037
>>
>> _______________________________________________
>> Wekalist mailing list -- [email protected]
>> Send posts to [email protected]
>> To unsubscribe send an email to [email protected]
>> To subscribe, unsubscribe, etc., visit 
>> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
>> List etiquette: 
>> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
> _______________________________________________
> Wekalist mailing list -- [email protected]
> Send posts to [email protected]
> To unsubscribe send an email to [email protected]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

-- 
PD Dr. Steffen Herbold
Institute of Computer Science, University of Goettingen
Goldschmidtstraße 7, 37077 Göttingen, Germany
mailto. [email protected]
tel. +49 551 39-172037

_______________________________________________
Wekalist mailing list -- [email protected]
Send posts to [email protected]
To unsubscribe send an email to [email protected]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html