Re: audiomixer usage

--=-qErfy2/vysgg1btLXF5a
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

That's an interesting topic, as the number of participants increases the pr=
oportion of redundant "participant-to-participant" mixing will also rise, e=
g with participants 1 2 3 4 you'll get such a set of combinations:

1 -> 2 3 4
2 -> 1 3 4
3 -> 1 2 4
4 -> 1 2 3

In this case we see that for instance the audio for participants 1 and 2 ge=
ts mixed twice together before being mixed with a final, different particip=
ant. (I'm sure there's a mathematical tool to calculate the amount of redun=
dancy here btw, if anyone knows?)

I suppose an application could construct with some care an arrangement of s=
equential audiomixers / tees to remove this redundancy.

As for getting audiomixer to support this optimization out of the box, the =
problem you'll encounter is that audiomixer is N sink pads to 1 source pad,=
 which doesn't lend itself well to such a design, you'd want a N to N audio=
mixer instead.

Cheers

On Thu, 2024-12-26 at 10:51 +0530, Pradeep Acharya via gstreamer-devel wrot=
e:
> Hi Michael,
>=20
> Thanks a lot for the quick reply . To keep the server load less, SFU is u=
sed for Video and MCU for audio so that the number of audio decoding that t=
he client has to do will be reduced . I was thinking of optimizing the numb=
er of audio mixers used in MCU architecture .From your explanation , i get =
that AEC=C2=A0 needs to be used to cut off participant's own audio. Again, =
this method would impose using AEC on each client.=C2=A0
>=20
> Regards =20
> Pradeep
>  =20
> On Wed, Dec 25, 2024 at 11:54=E2=80=AFPM Michael Gruner <[michael.gruner@=
ridgerun.com](mailto:[email protected])> wrote:
>=20
> > Hi there, happy holidays.
> > What you describe is a well known topology named MCU: Multipoint Contro=
l Unit. There are others worth exploring, like SFU: Selective Forwarding Un=
it or Mesh. I know this is not was what you asked, but I hope it eases your=
 research.
> >=20
> > [https://medium.com/@toshvelaga/webrtc-architectures-mesh-mcu-and-sfu-1=
2c502274d7](https://medium.com/@toshvelaga/webrtc-architectures-mesh-mcu-an=
d-sfu-12c502274d7)
> >=20
> > Now to your question, the MCU approach would be to have one audiomixer =
per client. Each client audiomixer will mix all the other channels, except =
itself. This is easy to implement, but will load tremendously the server. I=
t might not more manageable with audio only.
> >=20
> > The SFU approach would be not to use an audiomixer in the server at all=
. Each client would receive N network stream, each for every other clients.=
 These are then mixed locally by each client. This is very lightweight in t=
he server, but consumes exponentially more network bandwidth which, again, =
may be manageable with audio only.=C2=A0
> >=20
> > If, for whatever reason, you need to filter out using DSP techniques, y=
ou may look into echo cancellation. Check the work by Nicholas in
> >=20
> > [https://www.collabora.com/news-and-blog/blog/2016/07/08/gstreamer-echo=
-canceller/](https://www.collabora.com/news-and-blog/blog/2016/07/08/gstrea=
mer-echo-canceller/)
> >=20
> > Note that this is not quite the intended purpose and, personally, is no=
t the approach I would go for.
> >=20
> > Finally, please recall that you'll likely get more answers in the gstre=
amer discourse:
> >=20
> > [https://discourse.gstreamer.org/](https://discourse.gstreamer.org/)
> >  =20
> > Michael
> > [www.ridgerun.com](http://www.ridgerun.com)
> >  =20
> >=20
> > > On 25 Dec 2024, at 07:27, Pradeep Acharya via gstreamer-devel <[gstre=
[email protected]](mailto:[email protected].=
org)> wrote: =20
> > >=20
> >=20
> > > =EF=BB=BFHi All,
> > >=20
> > > I'm new to developing audio conferencing solutions on server side and=
 implementing the same using audiomixer plugin. The output of the mixer has=
 audio mixed that is received from all participants. The mixed audio has to=
 be transmitted to all participants.=C2=A0 Before transmitting mixed audio,=
 participants' audio has to be filtered out from the output generated by th=
e audio mixer and sent to that participant. Is there any plugin that checks=
 and filters out samples from the output generated by audiomixer plugin? If=
 there is no such plugin , kindly let me know other possible solutions for =
this.
> > >=20
> > > Thanks & Regards
> > >=20
> > >=20
> > >=20
> > >=20
> > > =20
> >
>=20

--=-qErfy2/vysgg1btLXF5a
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

<p>That's an interesting topic, as the number of participants increases the=
 proportion of redundant &quot;participant-to-participant&quot; mixing will=
 also rise, eg with participants 1 2 3 4 you'll get such a set of combinati=
ons:</p>
<p>1 -&gt; 2 3 4
2 -&gt; 1 3 4
3 -&gt; 1 2 4
4 -&gt; 1 2 3</p>
<p>In this case we see that for instance the audio for participants 1 and 2=
 gets mixed twice together before being mixed with a final, different parti=
cipant. (I'm sure there's a mathematical tool to calculate the amount of re=
dundancy here btw, if anyone knows?)</p>
<p>I suppose an application could construct with some care an arrangement o=
f sequential audiomixers / tees to remove this redundancy.</p>
<p>As for getting audiomixer to support this optimization out of the box, t=
he problem you'll encounter is that audiomixer is N sink pads to 1 source p=
ad, which doesn't lend itself well to such a design, you'd want a N to N au=
diomixer instead.</p>
<p>Cheers</p>
<p>On Thu, 2024-12-26 at 10:51 +0530, Pradeep Acharya via gstreamer-devel w=
rote:</p>
<blockquote type=3D"cite">
<p>Hi Michael,</p>
<p>Thanks a lot for the quick reply . To keep the server load less, SFU is =
used for Video and MCU for audio so that the number of audio decoding that =
the client has to do will be reduced . I was thinking of optimizing the num=
ber of audio mixers used in MCU architecture .From your explanation , i get=
 that AEC=C2=A0 needs to be used to cut off participant's own audio. Again,=
 this method would impose using AEC on each client.=C2=A0</p>
<p>Regards<br />
Pradeep</p>
<p>On Wed, Dec 25, 2024 at 11:54=E2=80=AFPM Michael Gruner &lt;<a href=3D"m=
ailto:[email protected]">[email protected]</a>&gt; wrot=
e:</p>
<blockquote type=3D"cite">
<p>Hi there, happy holidays.
What you describe is a well known topology named MCU: Multipoint Control Un=
it. There are others worth exploring, like SFU: Selective Forwarding Unit o=
r Mesh. I know this is not was what you asked, but I hope it eases your res=
earch.</p>
<p><a href=3D"https://medium.com/@toshvelaga/webrtc-architectures-mesh-mcu-=
and-sfu-12c502274d7">https://medium.com/@toshvelaga/webrtc-architectures-me=
sh-mcu-and-sfu-12c502274d7</a></p>
<p>Now to your question, the MCU approach would be to have one audiomixer p=
er client. Each client audiomixer will mix all the other channels, except i=
tself. This is easy to implement, but will load tremendously the server. It=
 might not more manageable with audio only.</p>
<p>The SFU approach would be not to use an audiomixer in the server at all.=
 Each client would receive N network stream, each for every other clients. =
These are then mixed locally by each client. This is very lightweight in th=
e server, but consumes exponentially more network bandwidth which, again, m=
ay be manageable with audio only.=C2=A0</p>
<p>If, for whatever reason, you need to filter out using DSP techniques, yo=
u may look into echo cancellation. Check the work by Nicholas in</p>
<p><a href=3D"https://www.collabora.com/news-and-blog/blog/2016/07/08/gstre=
amer-echo-canceller/">https://www.collabora.com/news-and-blog/blog/2016/07/=
08/gstreamer-echo-canceller/</a></p>
<p>Note that this is not quite the intended purpose and, personally, is not=
 the approach I would go for.</p>
<p>Finally, please recall that you'll likely get more answers in the gstrea=
mer discourse:</p>
<p><a href=3D"https://discourse.gstreamer.org/">https://discourse.gstreamer=
.org/</a></p>
<p>Michael
<a href=3D"http://www.ridgerun.com">www.ridgerun.com</a></p>
<blockquote type=3D"cite">
<p>On 25 Dec 2024, at 07:27, Pradeep Acharya via gstreamer-devel &lt;<a hre=
f=3D"mailto:[email protected]">[email protected]=
eedesktop.org</a>&gt; wrote:</p>
</blockquote>
<blockquote type=3D"cite">
<p>=EF=BB=BFHi All,</p>
<p>I'm new to developing audio conferencing solutions on server side and im=
plementing the same using audiomixer plugin. The output of the mixer has au=
dio mixed that is received from all participants. The mixed audio has to be=
 transmitted to all participants.=C2=A0 Before transmitting mixed audio, pa=
rticipants' audio has to be filtered out from the output generated by the a=
udio mixer and sent to that participant. Is there any plugin that checks an=
d filters out samples from the output generated by audiomixer plugin? If th=
ere is no such plugin , kindly let me know other possible solutions for thi=
s.</p>
<p>Thanks &amp; Regards</p>
</blockquote>
</blockquote>
</blockquote>

--=-qErfy2/vysgg1btLXF5a--