format spec unclear on how to quantize the LPC prediction

Hello!

I just noticed how the LPC prediction is quantized by browsing the
source code of the java port jFlac:

	:
	int sum = 0;
	for (int j = 0; j < order; j++)
		sum += qlpCoeff[j] * data[startAt+i-j-1];
	data[startAt + i] = residual[i] + (sum >> lpQuantization);
	:

I checked the C version (libFLAC) and it matches. However, it was a
surprise because this way of quantizing the prediction introduces a bias
towards negative infinity which results in residual samples that have a
bias towards positive infinity and therefore a shifted probability
distribution: residual of +1 is as common as 0 and more common than -1
etc...

This is not too bad I guess. But it CAN be accounted for -- for example
via an alternate entropy coding which uses a mapping from all possible
residual values to non-negative numbers by this interleaving:

	0	-> 0
	+1	-> 1
	-1	-> 2
	+2	-> 3
	-2	-> 4
	+3	-> 5
	etc

This way the symbols 0, 1, 2, ... have non-increasing probabilities.

Anyhow, this quantization issue is not mentioned in the format
specification. Naturally, I would have assumed to quantize by properly
rounding like this:

	data[startAt + i] = residual[i] + ((sum + a) >> lpQuantization);
where
	a = (1 << lpQuantization) / 2;

but I guess now it's too late to change that behaviour of the FIR predictor.


Cheers!
SG

_______________________________________________
Flac mailing list
[email protected]
http://lists.xiph.org/mailman/listinfo/flac

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHEc0ghc02VyxGQm0RAsLrAJ9NwFW4t+4pkmfS5lsQ2yc8ZfSFwgCeLKbb
F6eaVAfEx+y5lh1kp7heFuc=
=Dk8E
-----END PGP SIGNATURE-----