Access arguments relative to ESP in X32 mode?

A Debian maintainer reported a few issues for us under X32 (X32 is not
exactly X86, and its not exactly X64; see
https://wiki.debian.org/X32Port).

We are having trouble fixing a few of the issues because GDB is
available but doesn't quite work (see
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799556).

We have ASM code (both inline and block) that access arguments
relative to ESP. For example:

    #define SSE2_key_schedule(Y,X) \
        AS2(    movq    mm3, [X+7*8])\
        ...

        SSE2_key_schedule(esp+4, esi)

In the code above, the AS2 is just a two operand macro that abstracts
away the differences between Intel, AT&T and MASM assembly.

Under X32 with a non-corrupt stack, how would I access, say, what
formerly was located at ESP+4?

Thanks in advance.

(Sorry about asking X32 questions here. According to
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=692042, it looks
like they should got to debian-amd64@).