Unsynced Audio and Video signals in RTSP Stream

Dear GST Community,
i have implemented a rtsp server with the gst-rtsp-server module. The server receives an audio signal via an interaudiosrc and a video signal via an intervideosrc:
            intervideosrc channel=videosrc --> videoconvert --> vaapih264enc bitrate=5000 zerolatency=true --> rtph264pay pt=96 name=pay0
            interaudiosrc channel=audiosrc --> opusenc bitrate=64 --> queue --> rtpopuspay pt=97 name=pay1

When I try to view the stream via an rtsp client, for example VLC Player the stream works, but the audio signal is about 1 second behind the video signal. I already tried different audio encoders and different queue parameters like buffer size and leaky queues, but nothing solved the problem.
When I start the same pipeline as above but mux both signals and write them into a filesink they are synchronized.

I know that the server module internally uses the multiudpsink element. Is there any possibility to edit multiudpsink parameters from outside?
Does anybody have any ideas about this problem? I already read every google result I could find about it.

Best Regards,
Felix Sternkopf