Re: [f2py] MemoryLeak

Hi!

Since you are sure that f2py does not make copies then
it does not look like a memory leak due to f2py.

There could be several sources to this problem, for example,
a bug in numpy array reference counting, or old copies
of arrays are stored by the program and hence never released
during the loop, etc etc.

For a more detailed answer, could you prepare a minimal working script 
demonstrating the problem (it should demonstrate the memory increase 
during the program run).

Also, reusing existing arrays via inplace operations
during the loop might help.

To detect places that increase memory, you could insert
   print "line#", memusage()
to various places to the loop and learn something from the output.
(this assumes `from numpy.testing.utils import memusage` in the header
of the script)

Regards,
Pearu

Trond Kristiansen wrote:
> Hi all.
> 
> I have a problem with releasing memory from my Python+F2PY application. 
>  From Python, I loop over a list of large netCDF4 files, and for each 
> loop I extract a chunk of data (~200MB) which is stored in numpy arrays. 
> These data are then sent to Fortran functions wrapped using F2PY. This 
> works great for a while, before the application crashes with the error: 
> MemoryError (error code=12). I assume this is because for each time I 
> send an array to my Fortran function, a copy of the data is made which 
> is not released but builds up in memory. I can loop over 16 files before 
> my application crashes.
> 
> Since I assumed this was because of the extra copy of my array (a) done 
> in F2PY to convert from 'C' order to 'Fortran' order, I did a 
> "numpy.asarray(a,order='Fortran'). When I check my Fortran application 
> using -DF2PY_REPORT_ON_ARRAY_COPY I made sure that no copies were made.
> 
> However, my application still constantly increases in memory, and leaves 
> me with few clues to figure out what is going on. I call my function in 
> Fortran using expressions like the one shown below.
> 
>  From Python:
>  outdata = 
> interp.interpolation.dovertinter(np.asarray(outdata,order='Fortran'),
>                                                        
> np.asarray(data_ST,order='Fortran'),
>                                                        
> np.asarray(grdROMS.depth,order='Fortran'),
>                                                        
> np.asarray(grdROMS.z_r,order='Fortran'),
>                                                        
> np.asarray(grdSODA.z_r,order='Fortran'),
>                                                        
> int(grdROMS.Nlevels),
>                                                        
> int(grdSODA.Nlevels),
>                                                        int(grdROMS.xi_rho),
>                                                        
> int(grdROMS.eta_rho),
>                                                        int(grdROMS.xi_rho),
>                                                        
> int(grdROMS.eta_rho))
> 
> My Fortran routine looks like this:
> 
> double precision rz2, rz1
>             integer eta_rho, xi_rho, II, JJ, ic, jc, kc, kT, Nsoda, Nroms
>             double precision, dimension(Nsoda,JJ,II) :: dat
>             double precision, dimension(eta_rho,xi_rho) :: bathymetry
>             double precision, dimension(Nroms,JJ,II) :: outdat
>             double precision, dimension(Nsoda) ::  zs
>             double precision, dimension(Nroms,eta_rho,xi_rho) :: zr
> 
> !f2py intent(in) dat, bathymetry, zr, zs
> !f2py intent(in) Nroms, Nsoda, JJ, II, xi_rho, eta_rho
> !f2py intent(in,out) outdat
> !f2py intent(hide) ic,jc,kc,kT,rz1,rz2
> 
> 
> I hope that some of you with better knowledge of this can take a look 
> and give me a hint where it goes wrong.
> 
> Thank you for your help.
> 
> Cheers, Trond
> 
> 
> 
> _______________________________________________
> f2py-users mailing list
> f2py-users-Y4l6ocDipWCuvFJfX82//[email protected]
> http://cens.ioc.ee/mailman/listinfo/f2py-users