float->int rounding problem in setup_tone_curves()

"Martin Dufour" <[email protected]>
Newsgroups gmane.comp.multimedia.ogg.vorbis.devel
Message-ID <9383324BC3030A4E91CA2ABB865F63AA016E7301@SRV-SONIC.AudioKinetic.inte>
Hi,

We have severe audio quality issues when we compile Vorbis with
Microsoft VC 2005, while it works fine with VC 2003. I have isolated the
problem to a rounding issue in setup_tone_curves() ; the "lo_curve" and
"hi_curve" are different depending on the compiler. For example, here
are the values for the last band (#17) at 32 kHz prior to { ceil(),
floor() }:

VC2003: lo_fcurve=15.997361 hi_fcurve=16.000000 
VC2005: lo_fcurve:15.997360 hi_fcurve:15.999999

See how the floor() for hi_curve is problematic, as it will resolve to
15 or 16 depending on the compiler. This makes me think that perhaps a
round-to-nearest would be more appropriate (and indeed, seems to make
sense for other bands as well).

Why is hi_curve floored?

Thanks,
Martin Dufour
Audiokinetic
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.