Re: Re: transfrom dynamic range

>   residual [9 bits] -> transform -> [15 bits] -> quantize -> [X bits] -> inv quantize -> [15 bits] -> int transform -> [9 bits]
for this equation to be correct the inverse transform should be 
0.25   0.2   0.25    0.1
0.25   0.1  -0.25  -0.2
0.25  -0.1  -0.25   0.2
0.25  -0.2   0.25  -0.1
which isnt the case :(
and the f/i quant are not inverse of each other as well
please note that the result of the itrans is downscaled by 6   
So I think the correct chart should look something like that :
residual [9 bits] -> transform -> [15 bits] -> quantize -> [X bits] -> inv quantize -> [11 bits] -> int transform -> [16 bits] -> downscaling -> [9 bits]



<[email protected]>
Re: [Hdot264-devel] transfrom dynamic range
> The inverse, gains 4.
                    ^^^^
    This is correct?

    Let me recapitulate:

    residual [9 bits] -> transform -> [15 bits] -> quantize -> [X bits] -> inv quantize -> [15 bits] -> int transform -> [9 bits]

    The standard says that input to IT is in range [-2^15...( 2^15)-1] ( as Indrajit said), and output in [-2^15...( 2^15)-33].
Then, the normalization of EQ 8-278 takes it back to 9 bits.

    Another thing is that inside quantization ( using the reference software approach to change divisions by right shifts) dynamic
range go up to nearly 32 bit because of the multiplication with 14 bit constants.  As i see it, this could be problematic for optimized
MMX code unless both integer multiplication/divisions are used.

    /AITOR
----- Original Message ----- 
From: [email protected] 
To: [email protected] ; [email protected] 
Sent: Wednesday, April 02, 2003 12:56 PM
Subject: Re: [Hdot264-devel] transfrom dynamic range



The forward transfrom gaines 6 bits over the range of the input data ,
The inverse, gains 4.

<[email protected]>
[Hdot264-devel] transfrom dynamic range

Hi there!,

    Which is the real dynamic range of the values output by the forward transform?  Full 16 bit,
or even in a worst case less than that?

    /AITOR














Walla! Mail, Get Your Private, Free E-mail from Walla! at:
http://mail.walla.co.il

-----------------------------------------------------------------------
Walla! Mail, Get Your Private, Free E-mail from Walla! at:
http://mail.walla.co.il