[RFC] Dos and don'ts while removing multilib

Hi all,

I'm in the middle of preparing a patch in order to remove -m31 support
for s390.  After the removal, there is only 64-bit support left which
renders multilib obsolete on s390 ... or doesn't it?  In the past 32-bit
libraries were installed under /usr/lib and 64-bit libraries under
/usr/lib64 which I would like to keep.  Naively removing
MULTILIB_{OPTIONS,DIRNAMES,OSDIRNAMES} from Makefile
config/s390/t-linux64 and the define MULTILIB_DEFAULTS results in
choosing the wrong library dir `lib` instead of `lib64`.  For example,
on Fedora we would run into:

/devel/gcc-test/build/./gcc/xgcc -B/devel/gcc-test/build/./gcc/ -B/devel/gcc-test/dst/s390x-ibm-linux-gnu/bin/ -B/devel/gcc-test/dst/s390x-ibm-linux-gnu/lib/ -isystem /devel/gcc-test/dst/s390x-ibm-linux-gnu/include -isystem /devel/gcc-test/dst/s390x-ibm-linux-gnu/sys-include    -O2  -g -O2 -DIN_GCC   -W -Wall -Wno-error=narrowing -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition  -isystem ./include  -fPIC -mlong-double-128 -g -DIN_LIBGCC2 -fbuilding-libgcc -fno-stack-protector  -shared -nodefaultlibs -Wl,--soname=libgcc_s.so.1 -Wl,--version-script=libgcc.map  -o ./libgcc_s.so.1.tmp -g -O2 -B./ _muldi3_s.o _negdi2_s.o _lshrdi3_s.o _ashldi3_s.o _ashrdi3_s.o _cmpdi2_s.o _ucmpdi2_s.o _clear_cache_s.o _trampoline_s.o __main_s.o _absvsi2_s.o _absvdi2_s.o _addvsi3_s.o _addvdi3_s.o _subvsi3_s.o _subvdi3_s.o _mulvsi3_s.o _mulvdi3_s.o _negvsi2_s.o _negvdi2_s.o _ctors_s.o _ffssi2_s.o _ffsdi2_s.o _clz_s.o _clzsi2_s.o _clzdi2_s.o _ctzsi2_s.o _ctzdi2_s.o _popcount_tab_s.o _popcountsi2_s.o _popcountdi2_s.o _paritysi2_s.o _paritydi2_s.o _powisf2_s.o _powidf2_s.o _powixf2_s.o _powitf2_s.o _mulhc3_s.o _mulsc3_s.o _muldc3_s.o _mulxc3_s.o _multc3_s.o _divhc3_s.o _divsc3_s.o _divdc3_s.o _divxc3_s.o _divtc3_s.o _bswapsi2_s.o _bswapdi2_s.o _clrsbsi2_s.o _clrsbdi2_s.o _mulbitint3_s.o _fixunssfsi_s.o _fixunsdfsi_s.o _fixunsxfsi_s.o _fixsfdi_s.o _fixdfdi_s.o _fixxfdi_s.o _fixtfdi_s.o _fixunssfdi_s.o _fixunsdfdi_s.o _fixunsxfdi_s.o _fixunstfdi_s.o _floatdisf_s.o _floatdidf_s.o _floatdixf_s.o _floatditf_s.o _floatundisf_s.o _floatundidf_s.o _floatundixf_s.o _floatunditf_s.o _divdi3_s.o _moddi3_s.o _divmoddi4_s.o _udivdi3_s.o _umoddi3_s.o _udivmoddi4_s.o _udiv_w_sdiv_s.o _divmodbitint4_s.o _dpd_hf_to_sd_s.o _dpd_hf_to_dd_s.o _dpd_hf_to_td_s.o _dpd_sd_to_hf_s.o _dpd_dd_to_hf_s.o _dpd_td_to_hf_s.o sfp-exceptions_s.o extendhfsf2_s.o extendhfdf2_s.o extendhftf2_s.o truncsfhf2_s.o truncdfhf2_s.o trunctfhf2_s.o fixhfti_s.o fixunshfti_s.o floattihf_s.o floatuntihf_s.o floatbitinthf_s.o fixtfbitint_s.o floatbitinttf_s.o fixsfbitint_s.o floatbitintsf_s.o fixdfbitint_s.o floatbitintdf_s.o enable-execute-stack_s.o hardcfr_s.o strub_s.o unwind-dw2_s.o unwind-dw2-fde-dip_s.o unwind-sjlj_s.o unwind-c_s.o emutls_s.o libgcc.a -lc
/usr/bin/ld: cannot find crti.o: No such file or directory

Option -v gives a clue why this is failing:

LIBRARY_PATH=/devel/gcc-test/build/./gcc/:./:/lib/:/usr/lib/

The library path can be fixed via

MULTIARCH_DIRNAME = ../lib64$(call if_multiarch,:s390x-linux-gnu)

where we then have

LIBRARY_PATH=/devel/gcc-test/build/./gcc/:./:/lib/../lib64/:/lib/:/usr/lib/../lib64/:/usr/lib/

However, this works on Fedora but fails on Debian/Ubuntu with another
error:

make[2]: Entering directory '/devel/gcc-test/build/s390x-ibm-linux-gnu/libgcc'
# If this is the top-level multilib, build all the other
# multilibs.
/devel/gcc-test/build/./gcc/xgcc -B/devel/gcc-test/build/./gcc/ -B/devel/gcc-test/dst/s390x-ibm-linux-gnu/bin/ -B/devel/gcc-test/dst/s390x-ibm-linux-gnu/lib/ -isystem /devel/gcc-test/dst/s390x-ibm-linux-gnu/include -isystem /devel/gcc-test/dst/s390x-ibm-linux-gnu/sys-include    -g -O2 -O2  -g -O2 -DIN_GCC   -W -Wall -Wno-error=narrowing -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition  -isystem ./include  -fPIC -mlong-double-128 -g -DIN_LIBGCC2 -fbuilding-libgcc -fno-stack-protector  -fPIC -mlong-double-128 -I. -I. -I../.././gcc -I/devel/gcc-test/src/libgcc -I/devel/gcc-test/src/libgcc/. -I/devel/gcc-test/src/libgcc/../gcc -I/devel/gcc-test/src/libgcc/../include  -DHAVE_CC_TLS   -o _muldi3.o -MT _muldi3.o -MD -MP -MF _muldi3.dep -DL_muldi3 -c /devel/gcc-test/src/libgcc/libgcc2.c -fvisibility=hidden -DHIDE_EXPORTS
In file included from /devel/gcc-test/src/libgcc/../gcc/tsystem.h:95,
                 from /devel/gcc-test/src/libgcc/libgcc2.c:27:
/usr/include/stdio.h:28:10: fatal error: bits/libc-header-start.h: No such file or directory
   28 | #include <bits/libc-header-start.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Option -v shows:

#include <...> search starts here:
 .
 ../.././gcc
 /devel/gcc-test/src/libgcc
 /devel/gcc-test/src/libgcc/../gcc
 /devel/gcc-test/src/libgcc/../include
 /devel/gcc-test/build/./gcc/include
 /devel/gcc-test/build/./gcc/include-fixed
 /usr/local/include
 /usr/include
End of search list.

However, on Debian/Ubuntu we have a multiarch directory structure

$ ls /usr/include/bits
ls: cannot access '/usr/include/bits': No such file or directory
$ ls /usr/include/s390x-linux-gnu/bits/libc-header-start.h
/usr/include/s390x-linux-gnu/bits/libc-header-start.h

The only configuration which works so far on Fedora/Debian/Ubuntu is to
still enable multilib support via

config/s390/t-linux64:
MULTILIB_OPTIONS = m64
MULTILIB_DIRNAMES = 64
MULTILIB_OSDIRNAMES = ../lib64$(call if_multiarch,:s390x-linux-gnu)

config/s390/linux.h:
#define MULTILIB_DEFAULTS { "m64" }

However, having a multilib setup for a single option seems overly
complicated and slightly off to me.  Though, some comments  seem to
indicate that for some setups this maybe required.  Since I'm not
familiar with multilib/multiarch setups I might have missed something
and welcome any suggestions.  If everything fails, I will keep the
single multilib setup since this is basically the configuration which
works up to date.

Cheers,
Stefan