float->int rounding problem in setup_tone_curves()
"Martin Dufour" <[email protected]>
| Newsgroups | gmane.comp.multimedia.ogg.vorbis.devel |
|---|---|
| Message-ID | <9383324BC3030A4E91CA2ABB865F63AA016E7301@SRV-SONIC.AudioKinetic.inte> |
Hi,
We have severe audio quality issues when we compile Vorbis with
Microsoft VC 2005, while it works fine with VC 2003. I have isolated the
problem to a rounding issue in setup_tone_curves() ; the "lo_curve" and
"hi_curve" are different depending on the compiler. For example, here
are the values for the last band (#17) at 32 kHz prior to { ceil(),
floor() }:
VC2003: lo_fcurve=15.997361 hi_fcurve=16.000000
VC2005: lo_fcurve:15.997360 hi_fcurve:15.999999
See how the floor() for hi_curve is problematic, as it will resolve to
15 or 16 depending on the compiler. This makes me think that perhaps a
round-to-nearest would be more appropriate (and indeed, seems to make
sense for other bands as well).
Why is hi_curve floored?
Thanks,
Martin Dufour
Audiokinetic