Re: Video Coding Puzzle (fwd)

Hi,
	Mario sent along some sample vectors for this puzzle and compared
them to Matlab's idea of DCT. The numbers are close but not precise.

--
	-Mike Melanson


---------- Forwarded message ----------
Date: Sat, 10 Jul 2004 06:07:27 +0100
From: Mario Brito <[email protected]>
To: Mike Melanson <[email protected]>
Subject: Re: [codec-devel] Video Coding Puzzle (fwd)

[...]

> Also, if it's not too much trouble, do you think you could send a
> sample vector. I.e., a block of 64 samples before and after
> transformation?

Sure thing, no problem at all. I'll also take the opportunity and do something i
should have done long ago, which is to chew some of this data in MatLab and
compare the results.

I took two 8x8 matrices from two points in the code. The first is saved at the
beginning of FunctionA (in the pseudocode i wrote) with contents from the source
buffer and is labeled as the "input" matrix. The second is saved at the end of
FunctionB with contents of the destination buffer, and is labeled the "output"
matrix. Assuming FunctionA and FunctionB implement an IDCT (2D IDCT by calculating
two 1D IDCTs?), i should be able to run idct2 in matlab and reach a similar
result. I'm assuming the game is using an 18.14 fixed point format on the input
matrix, so i'm dividing the integers i get by 16384.0f before writing them to
file.

Case1:

Decoder Input:

820.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-106.700195 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-85.722656 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-61.233398 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-44.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-40.726563 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-51.737793 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-57.992188 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000

Decoder Output:

36 92 105 117 117 116 117 116
36 92 105 117 117 116 117 116
36 92 105 117 117 116 117 116
36 92 105 117 117 116 117 116
36 92 105 117 117 116 117 116
36 92 105 117 117 116 117 116
36 92 105 117 117 116 117 116
36 92 105 117 117 116 117 116

Matlab: --> matrix = idct2(input');

45.9998  109.8359   95.5582  120.9028  108.0977  115.1403  111.4655  112.9996
45.9998  109.8359   95.5582  120.9028  108.0977  115.1403  111.4655  112.9996
45.9998  109.8359   95.5582  120.9028  108.0977  115.1403  111.4655  112.9996
45.9998  109.8359   95.5582  120.9028  108.0977  115.1403  111.4655  112.9996
45.9998  109.8359   95.5582  120.9028  108.0977  115.1403  111.4655  112.9996
45.9998  109.8359   95.5582  120.9028  108.0977  115.1403  111.4655  112.9996
45.9998  109.8359   95.5582  120.9028  108.0977  115.1403  111.4655  112.9996
45.9998  109.8359   95.5582  120.9028  108.0977  115.1403  111.4655  112.9996


Case 2:

Input:

464.000000 37.489258 -12.246094 6.803711 -4.000000 0.000000 0.000000 0.000000
25.954102 -16.632813 -4.414063 -4.905273 0.000000 0.000000 -5.328613 0.000000
-52.045898 -13.242188 9.373047 0.000000 0.000000 0.000000 0.000000 0.000000
-34.018555 9.810547 0.000000 0.000000 0.000000 0.000000 6.285645 0.000000
-36.000000 0.000000 0.000000 3.401855 0.000000 0.000000 0.000000 0.000000
-35.635742 3.670410 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-36.955566 0.000000 0.000000 0.000000 7.391113 -9.407227 0.000000 0.000000
-28.996094 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000

Output:

36 67 69 74 77 73 65 56
38 72 75 76 75 69 60 50
38 75 74 76 74 66 57 43
38 76 78 75 70 60 49 39
39 74 77 72 65 56 44 41
39 74 70 66 59 49 44 42
38 76 67 58 47 35 37 38
37 70 64 47 37 32 30 35

Matlab:

   38.5001   76.1352   61.4000   72.8102   69.1894   68.3985   62.0658   55.5000
   39.5466   79.5930   67.8740   74.5255   66.7986   65.4940   57.2973   52.4490
   38.2815   81.3915   63.0527   73.4564   68.4323   64.5909   58.6469   46.7933
   37.8150   82.0237   66.8093   73.6516   64.6203   59.0312   51.9711   45.0757
   39.1841   77.7411   69.0237   70.6650   61.0644   59.2194   46.1807   47.9243
   37.5462   79.1405   60.0131   66.9803   59.7594   54.1424   49.6495   47.3781
   34.6244   82.8197   57.9975   63.1408   50.9082   41.0607   45.2362   43.3793
   34.4996   73.3878   57.3613   52.9761   43.0245   44.2959   36.9553   41.4999


Case3:

Input:

284.000000 25.954102 3.061523 -6.803711 -4.000000 0.000000 0.000000 0.000000
-2.883789 -2.079102 4.414063 0.000000 -2.883789 0.000000 5.328613 0.000000
-3.061523 -2.207031 -2.343262 2.603516 -3.061523 0.000000 0.000000 0.000000
-6.803711 2.452637 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-4.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
-7.391113 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
0.000000 0.000000 0.000000 12.329590 0.000000 0.000000 0.000000 0.000000

Output:

37 40 42 42 43 40 41 38
38 41 42 43 42 42 43 41
37 41 40 42 40 41 40 40
29 34 35 37 36 38 39 39
27 30 32 31 32 31 36 36
31 33 33 29 28 26 32 32
29 32 33 31 28 27 31 31
27 33 33 34 29 28 28 28

Matlab:

   36.4998   38.7791   41.6349   38.3203   42.6795   36.1933   41.3923   36.4999
   36.5981   41.0915   39.3498   41.7376   40.2722   40.3691   42.2160   40.4186
   37.0588   43.2327   37.7108   43.7328   36.6420   40.4696   38.6730   38.6525
   30.2957   36.3428   33.9088   38.7611   35.2325   37.5768   38.4246   37.9638
   28.7044   31.8879   33.6182   31.5808   34.4256   30.8960   37.3447   35.0361
   32.5270   33.7526   35.5467   29.0906   32.8777   26.2728   34.6849   31.7617
   29.8160   33.6903   33.2827   32.1118   31.5353   29.3415   34.3160   33.1673
   28.4999   35.7094   32.1192   37.2151   29.7853   31.7095   30.4623   30.5002


Well, the values aren't exactly the same... but are close enough. Differences are
likely due to compromises between decoding speed and precision.

I'll still try to figure out the specifics of how adding 3*2^51 ends up converting
a float to an int... Will look into the IEEE standard for some clues once i get
some sleep.

Take care,

Mario Brito


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com