Re: speex with jitter buffer
nomprenom <[email protected]> Mon, 25 Jan 2021 23:08:25 +0200
| Newsgroups | gmane.comp.audio.compression.speex.devel |
|---|---|
| Message-ID | <CABb1zCMybwtTKgkVOzpuMd+9TTB1S1E238oiWNPp0HN2ME3omA@mail.gmail.com> |
--===============7108827382409105264== Content-Type: multipart/alternative; boundary="0000000000000102fe05b9bff15e" --0000000000000102fe05b9bff15e Content-Type: text/plain; charset="UTF-8" Hello, so I finally managed to make it work. I answer here, maybe it could help someone in the future (I have also a small question at the end too) I don't use the speex_jitter_buffer.[ch] from the speex_client demo, I use directly the jitter.[ch] api instead. (There are few bugs in the implementation) I checked all the steps in my flow by saving the pcm or the speex data and listening to them before to continue. One of the problems was the high cpu usage (85%+) The process didn't have time to read on the pcm device sometimes : [pcm_read(40ms) + speex encoding] > 40 ms So, only with the encoding part, I started to lose frames... I found that passing the --enable-fixed-point to the compilation reduce the cpu usage from 85% to 15% (cpuinfo says there is an FPU on the platform, maybe it's not working well...) I was not using mutex arround the put and get of the jitter buffer like it is suggested in the doc. i fixed this too. My question is about the *jitter_buffer_tick()* function : when to call it on the player side ? I understood from the doc that it is the timer tick of the jitter buffer. - I tried without calling it too, there is no problem, why ? - I call it only if jitter_buffer_get() was OK. I thought, this way, if there is nothing in the jitter buffer, the timer will be "in pause" until something new will arrive, Is it the right way ? Thanks for your help. On Sun, 24 Jan 2021 at 18:08, nomprenom <[email protected]> wrote: > I did some changes: > - instead of "pcm size" units, i'm back to ms units > - I put in the buffer 40 ms directly instead of 2*20 > speex_jitter_put(&m_jitter, (char *)RFMessage->Data, SPEEX_SIZE_40MS, > CPSeqNumber * 40); > > On the player thread, I call 2 times speex_jitter_get, the first one with > a ts of 0, the 2nd one with 20ms timestamp offset - then I play the pcm > buffer. > I now wait for the first jitter_put before to do a jitter_get. > > So it seams to work now. > > The only problem is there are many warnings such as : > > warning: jitter buffer sees negative buffering, your code might be broken. > Value is -80 > warning: jitter buffer sees negative buffering, your code might be broken. > Value is -120 > warning: jitter buffer sees negative buffering, your code might be broken. > Value is -80 > warning: jitter buffer sees negative buffering, your code might be broken. > Value is -80 > etc.. > > How to fix this ? > > thanks > > On Sun, 24 Jan 2021 at 16:59, nomprenom <[email protected]> wrote: > >> Hi, >> I can't understand how to implement my encoder/decoder with the jitter >> buffer. I hope someone can help me: >> I have 2 threads. >> Because I don't have yet the full environment (with the devices etc..), >> - I'm reading from thread1, >> - sending to the main thread, >> - write to a jitter buff, >> - from th2 : read from the jitter and play the data. >> (Tomorrow, this is a device which will send the main thread the speex >> data via RF - that's why I need to handle a jitter). >> >> thread 1: >> 1) read 640 bytes from pcm (=40ms) (my settings are 8khz/16bits le) >> 2) encode to 40 bytes of speex >> 3) send to main thread 40 bytes of speex >> >> main thread: >> 1) speex_jitter_put(20 first bytes of speex, counter * 640) // timestamp >> = 0, 640, 1280, etc.. >> 2) speex_jitter_put(20 last bytes of speex, 320 + counter * 640) // >> timestamp = 320, 960, 1600, etc.. >> >> thread 2: >> 1) speex_jitter_get(buffer, NULL) >> 2) speex_jitter_get(buffer + 320, timestamp=320) >> 3) pcm_write(buffer, 640) >> >> Actually I can't hear anything. I tried many things without success. It's >> the most "stable" flow I found without seeing many warnings from the api >> (underrun, negative buffer, etc..) >> - in the main thread, I think speex_jitter_put always puts 20 ms, right ? >> (I initialized the buffer with an audio sampling rate of 8khz) >> - in the main thread, I don't know if it's ok to put directly 40ms of >> speex or if I need to put 2 * 20ms with a different timestamp for the 2nd >> - I don't know too well what the timestamp parameter is : is it a >> timestamp in ms ? I use now, after searching on the net, "pcm units" (320 = >> 20 ms/ 640 = 40ms) >> - in thread 2, I can't pcm_write 320 only else I see many "underrun" >> errors. I thought, at the beginning, to simply read from the jitter and >> play (320) and to do this 2 times. >> >> Any help will be appreciated. >> >> Thanks. >> >> >> >> >> On Mon, 18 Jan 2021 at 21:30, nomprenom <[email protected]> wrote: >> >>> Thank you Tristan for your answer. >>> >>> For now I took jitter.c and .h from dsp and added them to the makefile >>> of speex. >>> I also added speex_jitter_buffer.c and h from the sample into the lib. >>> >>> >>> >>> On Mon, 18 Jan 2021 at 20:30, Tristan Matthews <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> On Mon, Jan 18, 2021 at 12:29 PM nomprenom <[email protected]> wrote: >>>> > >>>> > Hi, >>>> > I'd like to use the speex encode/decoder library with the speex >>>> jitter buffer. >>>> > I took, first, the project "speex" from >>>> https://gitlab.xiph.org/xiph/speex >>>> > There is a nice sample in the directory speex_client/ which is using >>>> speex_jitter_buffer.h which itself is using <speex/speex_jitter.h> >>>> > I couldn't find this include file in the project. >>>> > >>>> > I checked in the project speexDSP and found it. >>>> > I thought, maybe, speexDsp is the new "speex" because it's deprecated. >>>> > >>>> > The problem is, now, with speexDsp I can't find the "speex.h" header >>>> and can't find either the speex_decode_init(), speex_decoder_ctl() APIs. >>>> > It's only in the sampledec.c in the doc/ directory of speexdsp >>>> > >>>> > How can I use the speex decoder/encoder with the speex_jitter api >>>> together ? >>>> >>>> The Speex project has been split in 2, speex "the codec" (where you'll >>>> find the decode_init and decoder_ctl API) and speexdsp "the DSP >>>> library" where you'll find the jitter buffer API. The motivation was >>>> that users want to be able to use purely the codec without the DSP >>>> stuff and vice-versa. >>>> >>>> N.B. The most active/used project in speexdsp is the resampler, you >>>> may find jitterbuffer implementations elsewhere that are a better fit >>>> for your project. >>>> >>>> See: >>>> https://gitlab.xiph.org/xiph/speex >>>> https://gitlab.xiph.org/xiph/speexdsp >>>> >>>> Best, >>>> Tristan >>>> >>> --0000000000000102fe05b9bff15e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hello, so I finally managed=C2=A0to make it work.<div>I an= swer here, maybe it could help someone=C2=A0in the future=C2=A0(I have also= a small question at the end too)</div><div><br><div>I don't use the sp= eex_jitter_buffer.[ch] from the speex_client demo, I use directly the jitte= r.[ch] api instead. (There are few bugs in the implementation)</div><div><b= r></div><div>I checked all the steps in my flow by saving the pcm or the sp= eex data and listening to them before to continue.</div><div><br></div><div= >One of the problems was the high cpu usage (85%+)=C2=A0 The process didn&#= 39;t have time to read=C2=A0on the pcm device sometimes : [pcm_read(40ms)= =C2=A0+ speex encoding] > 40 ms</div><div>So, only with the encoding=C2= =A0part, I started to lose frames...</div><div>I found that passing the=C2= =A0--enable-fixed-point to the compilation reduce the cpu usage from=C2=A08= 5% to 15% (cpuinfo says there is an FPU on the platform, maybe it's not= working well...)<br></div><div><br></div><div><div>I was not using mutex a= rround the put and get of the jitter buffer like it is suggested in the doc= . i fixed this too.</div><div></div></div><div><br></div><div>My question i= s about the=C2=A0<b>jitter_buffer_tick()</b> function : when to call it on = the player side ?</div><div>I understood from the doc that it is the timer = tick of the jitter buffer.</div><div><br></div><div>- I tried without calli= ng it too, there is no problem, why ?<br></div><div>- I call it only if=C2= =A0jitter_buffer_get() was OK. I thought, this way, if there is nothing in = the jitter buffer, the timer will be "in pause" until something n= ew will arrive, Is it the right way ?=C2=A0</div><div><br></div><div>Thanks= for your help.</div><div><br></div><div></div><div><br></div><div></div><d= iv><br></div></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" cl= ass=3D"gmail_attr">On Sun, 24 Jan 2021 at 18:08, nomprenom <<a href=3D"m= ailto:[email protected]">[email protected]</a>> wrote:<br></div><blo= ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left= :1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">I did some c= hanges:<div>- instead of "pcm size" units, i'm back to ms uni= ts</div><div>- I put in the buffer 40 ms directly instead of 2*20</div><div= >speex_jitter_put(&m_jitter, (char *)RFMessage->Data, SPEEX_SIZE_40M= S, CPSeqNumber * 40);<br></div><div><br></div><div>On the player thread, I = call 2 times=C2=A0speex_jitter_get, the first one with a ts of 0, the 2nd o= ne with 20ms timestamp offset - then I play the pcm buffer.</div><div>I now= wait for the first jitter_put before to do a jitter_get.</div><div><br></d= iv><div>So it seams to work now.</div><div><br></div><div>The only problem = is there are many warnings such as :</div><div><br></div><div>warning: jitt= er buffer sees negative buffering, your code might be broken. Value is =C2= =A0-80<br></div><div>warning: jitter buffer sees negative buffering, your c= ode might be broken. Value is =C2=A0-120<br></div><div>warning: jitter buff= er sees negative buffering, your code might be broken. Value is =C2=A0-80<b= r></div><div>warning: jitter buffer sees negative buffering, your code migh= t be broken. Value is =C2=A0-80<br></div><div>etc..</div><div><br></div><di= v>How to fix this ?</div><div><br></div><div>thanks</div></div><br><div cla= ss=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Sun, 24 Jan 202= 1 at 16:59, nomprenom <<a href=3D"mailto:[email protected]" target=3D"= _blank">[email protected]</a>> wrote:<br></div><blockquote class=3D"gm= ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,= 204,204);padding-left:1ex"><div dir=3D"ltr">Hi,<div>I can't understand = how to implement my encoder/decoder with the jitter buffer. I hope someone = can help me:</div><div>I have 2 threads.=C2=A0</div><div>Because I=C2=A0don= 't have yet the full environment (with the devices etc..),</div><div>- = I'm reading from thread1,=C2=A0</div><div>- sending to the main thread,= =C2=A0</div><div>- write to a jitter buff,=C2=A0</div><div>- from th2 : rea= d from the jitter and play the data.</div><div>(Tomorrow, this is a device = which will send the main thread the speex data via RF - that's why I ne= ed to handle a jitter).</div><div><br></div><div>thread 1:</div><div>1) rea= d 640 bytes from pcm (=3D40ms) (my settings are 8khz/16bits le)</div><div>2= ) encode to 40 bytes of speex</div><div>3) send to main thread 40 bytes of = speex</div><div><br></div><div>main thread:</div><div>1) speex_jitter_put(2= 0 first bytes of speex, counter * 640) // timestamp =3D 0, 640, 1280, etc..= </div><div>2) speex_jitter_put(20 last bytes of speex, 320 + counter * 640)= // timestamp =3D 320, 960, 1600, etc..</div><div><br></div><div>thread 2:<= /div><div>1)=C2=A0speex_jitter_get(buffer, NULL)</div><div>2)=C2=A0 speex_jitter_get(buffer + 320, timestamp=3D320) </div><div>3) pcm_write(buffer, 640)</div><div><br></div><div>Actually=C2= =A0I can't hear anything. I tried many things without success. It's= the most "stable" flow I found without seeing many warnings from= the api (underrun, negative buffer, etc..)</div><div>- in the main thread,= I think speex_jitter_put always=C2=A0puts 20 ms, right ? (I initialized th= e buffer=C2=A0with an audio sampling rate of 8khz)</div><div>- in the main = thread, I don't know if it's ok to put directly 40ms of speex or if= I need to put 2 * 20ms with a different timestamp for the 2nd</div><div>- = I don't know too well what the timestamp parameter is : is it a timesta= mp in ms ? I use now, after searching on the net, "pcm units" (32= 0 =3D 20 ms/ 640 =3D 40ms)=C2=A0</div><div>- in thread 2, I can't pcm_w= rite 320 only else I see many "underrun" errors. I thought, at th= e beginning, to simply read from the jitter and play (320) and to do this 2= times.</div><div><br></div><div>Any help will be appreciated.</div><div><b= r></div><div>Thanks.</div><div><br></div><div><br></div><div><br></div></di= v><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On M= on, 18 Jan 2021 at 21:30, nomprenom <<a href=3D"mailto:[email protected]= om" target=3D"_blank">[email protected]</a>> wrote:<br></div><blockquo= te class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">Thank you=C2=A0Tr= istan for your answer.<div><br><div>For now I took jitter.c and .h from dsp= and added them to the makefile of speex.</div><div>I also added speex_jitt= er_buffer.c and h from the sample into the lib.<br></div><div><br></div><di= v><br></div></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" cla= ss=3D"gmail_attr">On Mon, 18 Jan 2021 at 20:30, Tristan Matthews <<a hre= f=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>&= gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0= px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<b= r> <br> On Mon, Jan 18, 2021 at 12:29 PM nomprenom <<a href=3D"mailto:nomprenom@= gmail.com" target=3D"_blank">[email protected]</a>> wrote:<br> ><br> > Hi,<br> > I'd like to use the speex encode/decoder library with the speex ji= tter buffer.<br> > I took, first, the project "speex" from <a href=3D"https://g= itlab.xiph.org/xiph/speex" rel=3D"noreferrer" target=3D"_blank">https://git= lab.xiph.org/xiph/speex</a><br> > There is a nice sample in the directory speex_client/ which is using s= peex_jitter_buffer.h which itself is using <speex/speex_jitter.h><br> > I couldn't find this include file in the project.<br> ><br> > I checked in the project speexDSP and found it.<br> > I thought, maybe, speexDsp is the new "speex" because it'= ;s deprecated.<br> ><br> > The problem is, now, with speexDsp I can't find the "speex.h&= quot; header and can't find either the speex_decode_init(), speex_decod= er_ctl() APIs.<br> > It's only in the sampledec.c in the doc/ directory of speexdsp<br> ><br> > How can I use the speex decoder/encoder with the speex_jitter api toge= ther ?<br> <br> The Speex project has been split in 2, speex "the codec" (where y= ou'll<br> find the decode_init and decoder_ctl API) and speexdsp "the DSP<br> library" where you'll find the jitter buffer API. The motivation w= as<br> that users want to be able to use purely the codec without the DSP<br> stuff and vice-versa.<br> <br> N.B. The most active/used project in speexdsp is the resampler, you<br> may find jitterbuffer implementations elsewhere that are a better fit<br> for your project.<br> <br> See:<br> <a href=3D"https://gitlab.xiph.org/xiph/speex" rel=3D"noreferrer" target=3D= "_blank">https://gitlab.xiph.org/xiph/speex</a><br> <a href=3D"https://gitlab.xiph.org/xiph/speexdsp" rel=3D"noreferrer" target= =3D"_blank">https://gitlab.xiph.org/xiph/speexdsp</a><br> <br> Best,<br> Tristan<br> </blockquote></div> </blockquote></div> </blockquote></div> </blockquote></div> --0000000000000102fe05b9bff15e-- --===============7108827382409105264== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KU3BlZXgtZGV2 IG1haWxpbmcgbGlzdApTcGVleC1kZXZAeGlwaC5vcmcKaHR0cDovL2xpc3RzLnhpcGgub3JnL21h aWxtYW4vbGlzdGluZm8vc3BlZXgtZGV2Cg== --===============7108827382409105264==--