Re: mipsel out-of-memory building dolfinx 0.4.0

On Sun, Jun 12, 2022 at 5:38 PM Drew Parsons <[email protected]> wrote:
>
> Hi MIPS porters, I need helping with the build of fenics-dolfinx on
> mipsel from experimental,
> https://salsa.debian.org/science-team/fenics/fenics-dolfinx/-/tree/experimental
>
> It previously built fine, but the latest version 1:0.4.1 has evidently
> grown large enough to break on mipsel.
>
> mip64el builds successfully, as we might expect. But it's not simply a
> 32-bit issue in the sense that i386, armel, armhf are all still building
> successfully.
>
> Build logs can be found at
> https://buildd.debian.org/status/logs.php?pkg=fenics-dolfinx&arch=mipsel
>
> In the build for 1:0.4.1-1exp1 we see:
>
> [ 83%] Building CXX object CMakeFiles/cpp.dir/dolfinx/wrappers/nls.cpp.o
> /usr/bin/c++ -DBOOST_ALL_NO_LIB -DBOOST_CHRONO_DYN_LINK
> -DBOOST_TIMER_DYN_LINK -DDOLFINX_VERSION=\"0.4.1\" -DHAS_PTSCOTCH
> -DHAS_SLEPC -D_FILE_OFFSET_BITS=64 -Dcpp_EXPORTS
> -I/usr/lib/petscdir/petsc3.16/mipsel-linux-gnu-complex/lib/python3/dist-packages/petsc4py/include
> -I/usr/lib/python3/dist-packages/mpi4py/include -isystem
> /usr/include/python3.9 -isystem
> /usr/lib/python3/dist-packages/pybind11/include -isystem
> /usr/lib/python3/dist-packages/ffcx/codegeneration -isystem
> /usr/lib/mipsel-linux-gnu/openmpi/include -isystem
> /usr/lib/mipsel-linux-gnu/openmpi/include/openmpi -isystem
> /usr/lib/petscdir/petsc3.16/mipsel-linux-gnu-complex/include -isystem
> /usr/include/hdf5/openmpi -isystem
> /usr/lib/slepcdir/slepc3.16/mipsel-linux-gnu-complex/include -g -O2
> -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat
> -Werror=format-security -isystem
> /<<PKGBUILDDIR>>/debian/tmp-complex/usr/include -DVERSION_INFO=\"0.4.1\"
> -O3 -DNDEBUG -fPIC -fvisibility=hidden -Wall -Werror -pedantic -flto
> -fno-fat-lto-objects -pthread -Wdate-time -D_FORTIFY_SOURCE=2 -std=c++17
> -MD -MT CMakeFiles/cpp.dir/dolfinx/wrappers/nls.cpp.o -MF
> CMakeFiles/cpp.dir/dolfinx/wrappers/nls.cpp.o.d -o
> CMakeFiles/cpp.dir/dolfinx/wrappers/nls.cpp.o -c
> /<<PKGBUILDDIR>>/python/dolfinx/wrappers/nls.cpp
> [ 91%] Building CXX object
> CMakeFiles/cpp.dir/dolfinx/wrappers/refinement.cpp.o
> /usr/bin/c++ -DBOOST_ALL_NO_LIB -DBOOST_CHRONO_DYN_LINK
> -DBOOST_TIMER_DYN_LINK -DDOLFINX_VERSION=\"0.4.1\" -DHAS_PTSCOTCH
> -DHAS_SLEPC -D_FILE_OFFSET_BITS=64 -Dcpp_EXPORTS
> -I/usr/lib/petscdir/petsc3.16/mipsel-linux-gnu-complex/lib/python3/dist-packages/petsc4py/include
> -I/usr/lib/python3/dist-packages/mpi4py/include -isystem
> /usr/include/python3.9 -isystem
> /usr/lib/python3/dist-packages/pybind11/include -isystem
> /usr/lib/python3/dist-packages/ffcx/codegeneration -isystem
> /usr/lib/mipsel-linux-gnu/openmpi/include -isystem
> /usr/lib/mipsel-linux-gnu/openmpi/include/openmpi -isystem
> /usr/lib/petscdir/petsc3.16/mipsel-linux-gnu-complex/include -isystem
> /usr/include/hdf5/openmpi -isystem
> /usr/lib/slepcdir/slepc3.16/mipsel-linux-gnu-complex/include -g -O2
> -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat
> -Werror=format-security -isystem
> /<<PKGBUILDDIR>>/debian/tmp-complex/usr/include -DVERSION_INFO=\"0.4.1\"
> -O3 -DNDEBUG -fPIC -fvisibility=hidden -Wall -Werror -pedantic -flto
> -fno-fat-lto-objects -pthread -Wdate-time -D_FORTIFY_SOURCE=2 -std=c++17
> -MD -MT CMakeFiles/cpp.dir/dolfinx/wrappers/refinement.cpp.o -MF
> CMakeFiles/cpp.dir/dolfinx/wrappers/refinement.cpp.o.d -o
> CMakeFiles/cpp.dir/dolfinx/wrappers/refinement.cpp.o -c
> /<<PKGBUILDDIR>>/python/dolfinx/wrappers/refinement.cpp
>
> cc1plus: out of memory allocating 17707045 bytes after a total of
> 42041344 bytes
>
>
> In the first instance I tried switching off LTO, adding the cmake
> configuration "set(CMAKE_INTERPROCEDURAL_OPTIMIZATION FALSE)" which
> deactivates -flto.
> But it didn't fix the problem.  The corresponding log for 1:0.4.1-1exp2
> shows much the same problem,
>
>    virtual memory exhausted: Cannot allocate memory
>
> I then tried building on the porterbox eller.debian.org, reducing
> optimization from -O3  to -O2, but that gave the same result.
>
> Finally I tried with no optimizations at all (-O0) via cmake option
> add_compile_options(-O0).  This did allow the object files to build, but
> then linking failed with relocation truncated:
>
> [100%] Linking CXX shared module
> ../lib.linux-mips-3.10/dolfinx/cpp.cpython-310-mipsel-linux-gnu.so
> /usr/bin/cmake -E cmake_link_script CMakeFiles/cpp.dir/link.txt
> --verbose=1
> /usr/bin/c++ -fPIC -g -O2
> -ffile-prefix-map=/home/dparsons/dolfinx/fenics-dolfinx=.
> -fstack-protector-strong -Wformat -Werror=format-security -isystem
> /home/dparsons/dolfinx/fenics-dolfinx/debian/tmp-complex/usr
> /include -DVERSION_INFO=\"0.4.1\" -O3 -DNDEBUG -pthread -Wl,-z,relro
> -shared  -o
> ../lib.linux-mips-3.10/dolfinx/cpp.cpython-310-mipsel-linux-gnu.so
> CMakeFiles/cpp.dir/dolfinx/wrappers/dolfinx.cpp.o CMakeFiles/cp
> p.dir/dolfinx/wrappers/common.cpp.o
> CMakeFiles/cpp.dir/dolfinx/wrappers/fem.cpp.o
> CMakeFiles/cpp.dir/dolfinx/wrappers/geometry.cpp.o
> CMakeFiles/cpp.dir/dolfinx/wrappers/graph.cpp.o
> CMakeFiles/cpp.dir/dolfinx/wra
> ppers/io.cpp.o CMakeFiles/cpp.dir/dolfinx/wrappers/la.cpp.o
> CMakeFiles/cpp.dir/dolfinx/wrappers/log.cpp.o
> CMakeFiles/cpp.dir/dolfinx/wrappers/mesh.cpp.o
> CMakeFiles/cpp.dir/dolfinx/wrappers/nls.cpp.o CMakeFiles/c
> pp.dir/dolfinx/wrappers/refinement.cpp.o
> -Wl,-rpath,/home/dparsons/dolfinx/fenics-dolfinx/debian/tmp-complex/usr/lib/mipsel-linux-gnu:/usr/lib/mipsel-linux-gnu/openmpi/lib:/usr/lib/mipsel-linux-gnu/hdf5/openmpi
> :/usr/lib/slepcdir/slepc3.16/mipsel-linux-gnu-complex/lib:/usr/lib/petscdir/petsc3.16/mipsel-linux-gnu-complex/lib
> /home/dparsons/dolfinx/fenics-dolfinx/debian/tmp-complex/usr/lib/mipsel-linux-gnu/libdolfinx_com
> plex.so.0.4.1 /usr/lib/mipsel-linux-gnu/libbasix.so.0.4.2
> /usr/lib/mipsel-linux-gnu/libboost_timer.so.1.74.0
> /usr/lib/mipsel-linux-gnu/libboost_chrono.so.1.74.0
> /usr/lib/mipsel-linux-gnu/openmpi/lib/libmpi_cxx.so
> /usr/lib/mipsel-linux-gnu/openmpi/lib/libmpi.so
> /usr/lib/mipsel-linux-gnu/hdf5/openmpi/libhdf5.so
> /usr/lib/slepcdir/slepc3.16/mipsel-linux-gnu-complex/lib/libslepc_complex.so
> /usr/lib/petscdir/petsc3.16/mipsel-linux-gnu-complex/lib/libpetsc_complex.so
> CMakeFiles/cpp.dir/dolfinx/wrappers/fem.cpp.o: in function
> `_PyObject_TypeCheck':
> /usr/include/python3.10/object.h:247:(.text+0xa0): relocation truncated
> to fit: R_MIPS_CALL16 against `PyType_IsSubtype'
> CMakeFiles/cpp.dir/dolfinx/wrappers/fem.cpp.o: in function `_Py_DECREF':
> /usr/include/python3.10/object.h:500:(.text+0x174): relocation truncated
> to fit: R_MIPS_CALL16 against `_Py_Dealloc'
> CMakeFiles/cpp.dir/dolfinx/wrappers/fem.cpp.o: in function
> `__Pyx_ImportFunction(_object*, char const*, void (**)(), char const*)':
> /usr/lib/python3/dist-packages/mpi4py/include/mpi4py/mpi4py.MPI_api.h:111:(.text+0x46c):
> relocation truncated to fit: R_MIPS_CALL16 against
> `PyObject_GetAttrString'
> /usr/lib/python3/dist-packages/mpi4py/include/mpi4py/mpi4py.MPI_api.h:114:(.text+0x498):
> relocation truncated to fit: R_MIPS_CALL16 against
> `PyDict_GetItemString'
> /usr/lib/python3/dist-packages/mpi4py/include/mpi4py/mpi4py.MPI_api.h:116:(.text+0x4bc):
> relocation truncated to fit: R_MIPS_GOT16 against `PyExc_ImportError'
> /usr/lib/python3/dist-packages/mpi4py/include/mpi4py/mpi4py.MPI_api.h:116:(.text+0x4c8):
> relocation truncated to fit: R_MIPS_CALL16 against `PyModule_GetName'
> /usr/lib/python3/dist-packages/mpi4py/include/mpi4py/mpi4py.MPI_api.h:116:(.text+0x4f0):
> relocation truncated to fit: R_MIPS_CALL16 against `PyErr_Format'
> /usr/lib/python3/dist-packages/mpi4py/include/mpi4py/mpi4py.MPI_api.h:122:(.text+0x514):
> relocation truncated to fit: R_MIPS_CALL16 against `PyCapsule_IsValid'
> /usr/lib/python3/dist-packages/mpi4py/include/mpi4py/mpi4py.MPI_api.h:123:(.text+0x538):
> relocation truncated to fit: R_MIPS_GOT16 against `PyExc_TypeError'
> /usr/lib/python3/dist-packages/mpi4py/include/mpi4py/mpi4py.MPI_api.h:123:(.text+0x544):
> relocation truncated to fit: R_MIPS_CALL16 against `PyModule_GetName'
> /usr/lib/python3/dist-packages/mpi4py/include/mpi4py/mpi4py.MPI_api.h:123:(.text+0x560):
> additional relocation overflows omitted from the output
> collect2: error: ld returned 1 exit status
> gmake[4]: *** [CMakeFiles/cpp.dir/build.make:266:
> ../lib.linux-mips-3.10/dolfinx/cpp.cpython-310-mipsel-linux-gnu.so]
> Error 1
>
>
> At this point I'm out of clues.  What do you recommend?

I used to see something similar when building Crypto++ on resource
constrained dev-boards without swap files, like a BeagleBoard. cc1plus
used to crash all the time when certain source files were built,
especially ones that included a lot of header files. The symptom was
the same as:

> cc1plus: out of memory allocating 17707045
> bytes after a total of 42041344 bytes

Drop back to 1 make job. Don't use `make -j N`. Instead, just use
`make` or `make -j 1`.

Jeff