Re: get duplicate frames when doing multiprocessing

Yodash wrote:
>
> my target is to load data frames in parallel in order to save time 
> (trigger for the action is button click on frame)
> you are correct, i loaded same df 3 times but its just an example (on 
> my production app they are not the same)
>
> the issue is that whenever I try to perform parallelism using 
> multiprocessing..my frame is duplicated as number of child processes I 
> declare
>
> it will be a pity not to be able use parallelism from wx...

Well, this is not really a wx issue at all.  You'd have the same basic 
issues loading these pandas frames in a non-GUI app.  It's just very 

easy to forget about what environment you're running in.

Think about how multiprocessing works.  Your code launches a new 
process, which loads a new copy of the Python interpreter, and then has 
to re-import all of the modules your code needs.  Pandas loads Numpy, 
and both of those modules are very, very large.  It can take 3 or 4 
seconds just to import those modules, and you're going to pay that same 
penalty in every subprocess  you launch. Unless your data import takes 
considerably longer than that, multiprocessing is not going to be a win.

If your data does take a very long time, then you can use a Queue to 
ship the constructed dataframes back to the mothership.

-- 
Tim Roberts, [email protected]
Providenza & Boekelheide, Inc.


-- 
You received this message because you are subscribed to the Google Groups "wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/wxpython-users/64e42b28-6c69-8500-c0be-f9e76b1a3f03%40probo.com.