[f2py] F2PY: Unable to prevent Fortran ordered array copy

Hi,

I have been using Numpy+f2py for a couple of years, and the combination 
  has worked well for me for many types of problems.  However, I have 
never been able to prevent F2PY from copying arrays even though I 
initialize them as Fortran ordered in the calling python script.  I have 
spent a considerable time looking in the archives (both F2PY & NUMPY) 
and found several leads to diagnosing array-copy problems (see results 
below), but none of them seem to work for me.

-----------------------
System/Compiler Info:
-----------------------
I am currently running on a Dell workstation with the following setup:
[Fedora Core 3 Linux (32-bit)]/
[Python:2.5.1]/[Numpy:1.0.4]/[f2py:2.45.241_1926]/
[F90-compiler:Intel 10.0].

Actually, I have tried wrapping using three different compilers: 
INTEL-F90 (IFORT, v.10.0), PGF90 (v.8.0), and GFORTRAN (gcc 4.2.0).  But 
array-copying persists in every case.

For reference, I have uploaded a TGZ file containing 7 text files to the 
following location (the filelist - test sources, PYF, and output - are 
summarized below this email): 
ftp://ftp.gps.caltech.edu/pub/rkanda/MemoryDeallocTest.tgz.  If 
attachments are a preferred way to send files, please do let me know.

The problem details and test results are as follows ...

----------------------
Problem description:
----------------------
I've to call a wrapped f90 function several times from the same python 
script, each time with different array dimensions (please see 'mm.f90', 
and 'call_mm.py' in TGZ file).  I want to be able to handle arrays of 
size 10000x10000 (1e8 elements).  I am finding that after each call, 
memory allocated during f2py array-copy (which I haven't been able to 
eliminate) is not being deallocated, and I find (using 
numpy.testing.utils.memusage()) that the additional memory usage after 
each call is about the same as the size of all the F2PY-copied-arrays 
for that call (see 'call_mm.py').  So, the virtual memory usage 
mushrooms, and when it gets to ~2GB (I have 1.5 GB of RAM, and 6 GB of 
virtual memory on my system), I encounter a "Core Dump".  [As an aside, 
I am not sure whether the core-dump is due to a "per-process-limit" for 
Python or the 32-bit Linux OS].  I am using the "-DF2PY_REPORT_ATEXIT" 
and "-DF2PY_REPORT_ON_ARRAY_COPY=1" options on the f2py command line to 
document the array copying (see wrapPYF.csh & SampleOutput*.txt files in 
TGZ file).

--------------------------
Summary of observations:
--------------------------
1) I have tried wrapping using three different compilers: INTEL-F90 
(IFORT, v.10.0), PGF90 (v.8.0), and GFORTRAN (gcc 4.2.0).  But 
array-copying persists in every case.

--------------------------
2) The wrapper generated using 'intent(in,out,overwrite)' "works", but 
creates a copy every time (see 'mm_new.pyf' & 
SampleOutput*_InOutOverwrite.txt in TGZ file).  I also tried by 
explicitly setting the 'overwrite_<VAR>' flags to 1, just to be sure. 
If I test with a constant array size (i.e., disable array size update 
during each iteration), then I see that arrays are being overwritten for 
every iteration AFTER the first.  So, 'intent(in,out,overwrite)' seems 
to do the job except for the first array copy.
	On the other hand, intent(inout), or intent(inplace) do not work, and I 
get the following run-time error: "pymm.error: failed in converting 4th 
argument `a' of pymm.mmfunc.mm to C/Fortran array" (see 
SampleOutput*_InPlace.txt for details).

--------------------------
3) I have tried defining the numpy array in fortran ordering several 
ways (order keyword in array or asarray, asfortranarray; see 
'call_mm.py'), but all of them result in arrays being copied before 
being passed to the wrapped function (pymm.mmfunc.mm).

--------------------------
4) I tested that my numpy installation was working correctly:
 >>>
 >>> a = reshape(arange(15),(3,5))
 >>> a
array([[ 0,  1,  2,  3,  4],
         [ 5,  6,  7,  8,  9],
         [10, 11, 12, 13, 14]])
 >>> b = reshape(arange(15),(3,5),order='f')
 >>> b
array([[ 0,  3,  6,  9, 12],
         [ 1,  4,  7, 10, 13],
         [ 2,  5,  8, 11, 14]])
 >>> isfortran(a)
False
 >>> a.flags['F_CONTIGUOUS']
False
 >>> isfortran(b)
True
 >>> b.flags['F_CONTIGUOUS']
True
 >>>

--------------------------
5) I use a similar test procedure to (4) in the attached python script, 
'call_mm.py'.  But right after these checks, when I run 
"pymm.has_column_major_storage(a)" (pymm is the wrapped module, and 'a' 
the array), however, the result is '0' (ZERO - i.e, not an f-contiguous 
array)!  I don't understand why f2py is not recognizing a 
fortran-ordered array assigned within numpy.  I am guessing 
'intent(inplace)' or 'intent(inout)' in (1) are not working for me 
because of this same reason - f2py does not seem to recognize the input 
array as being fortran-ordered.  Also see (6) below.

--------------------------
6) I also tried converting all input arrays before calling the wrapped 
function (pymm) using "a = pymm.as_column_major_storage(a)", but that 
gives me an error.  To illustrate, for the array defined in (4) above, 
this function fails as follows:
 >>>
 >>> a
array([[ 0,  1,  2,  3,  4],
         [ 5,  6,  7,  8,  9],
         [10, 11, 12, 13, 14]])
 >>> pymm.has_column_major_storage(a)
0
 >>> a = pymm.as_column_major_storage(a)
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
pymm.error: expected array object
 >>>

--------------------------
Given that there are several previous posts where one of the above seems 
to have worked for others, it seems like there is something I don't 
understand about f2py array passing, or f2py directives in the signature 
file.  Perhaps there is a subtle problem with my f2py installation.

I am including all my test-scripts & output files (along with the f2py 
command used to wrap the f90 code & wrapping log) at the location below. 
  I will really appreciate any help/suggestions in this regard.

Thank you very much in advance for your time.
Ravi.

================================================================
Test file summary (the following files can be downloaded from:
ftp://ftp.gps.caltech.edu/pub/rkanda/MemoryDeallocTest.tgz)
----------------------------------------------------------------
1. mm.f90     : Fortran test function
2. mm_new.pyf : Edited signature file for 'mm.f90'
3. wrapPYF.csh: C-Shell script to wrap 'mm.f90', using 'mm_new.pyf'
4. mm_f2pywrap.log: F2PY compile screen output
5. call_mm.py : Python script calling the wrapped 'mm.f90'
6. Sample*_InOutOverwrite.txt: Screen output from 'call_mm.py' for 
'intent(in,out,overwrite)' case.
7. Sample*_InPlace.txt: Screen output from 'call_mm.py' for 
'intent(inplace)' case.
=================================================================