Re: Beginner Hurdles

Hi Zack,
I think I can help with some of this since I wrote the ogg/theora
decoder for LiVES (lives.sf.net) and more recently I reworked the
ogg/theora demuxer for vlc.

Converting from yuv420 to rgb:
take a look at the code here
http://lives.svn.sourceforge.net/viewvc/lives/trunk/src/colourspace.c
convert_yuv420p_to_rgb_frame()

for theora you should use clamped yuv (16-235, 16-240, 16-240). The
function uses arrays for conversion and does chroma super sampling
using an averaging array.


Seeking back to the beginning:
you can seek to byte 0 and skip the bos packets, or you can store the
position of data_start which follows the bos packets and seek back to
that. You must remember to call ogg_stream_reset() for each logical
stream after doing any seek operation.



Regards,
Salsaman.

http://lives.sourceforge.net
https://www.ohloh.net/accounts/salsaman




On Sun, Aug 15, 2010 at 5:30 PM, Zack Morris <[email protected]> wrote:
> Hey everyone, I just got Theora running on my Mac, and ran across
> several hurdles, that I was wondering if someone could help me with.
> I do a lot of tech support at work, and get the same questions over
> and over, so I tried skimming the archives but couldn't find the
> answers.  Maybe these could go in a FAQ of some sort?  These are
> fairly unavoidable issues that should probably be better documented in
> the example at http://svn.xiph.org/trunk/theora/examples/player_example.c
>  since I think most people are looking for something a little higher
> level.  Here they are:
>
> 1. Many of us just want to render each frame to RGB. I decided to skip
> SDL for now and render the RGB in software so that I understand how it
> works.  I will probably make a shader to render the YUV directly at
> some point, but speed is not the issue for me right now.
>
> I'm rendering a 4:2:0 movie by hand right now, stretching the UV to
> double size like the Y.  If you follow the example on the line that
> says "} else if (px_fmt==TH_PF_444){" to convert YUV to RGB, you will
> end up with reds that are salmon colored, and the blues are too navy
> colored.  I've posted an example here:
>
> http://www.postimage.org/image.php?v=TsBWFfr
>
> The top 3 frames are variations where I'm trying to align to UV
> channels, but the important thing is that VLC does a very good job
> matching colors, and even though MPlayer falls down on the rendering,
> they both do much better than my code.  I'm guessing that the example
> code suffers from some problem with gamma on the Mac, so I'm wondering
> if someone has the proper coefficients to render:
>
> r = (1904000*(*py)+2609823*(*pv)-363703744)/1635200;
> g = (3827562*(*py)-1287801*(*pu)-2672387*(*pv)+447306710)/3287200;
> b = (952000*(*py)+1649289*(*pu)-225932192)/817600;
>
> Or if they have the conversion that will stretch this RGB to the right
> color space for the Mac.  I'm mainly concerned with the red and blue
> channels.  it's ok if the linearity is incorrect, because I'm mostly
> concerned with the max and min values.
>
> If nobody has a solution, then I will just sample my colors and come
> up with the factor to scale them to VLC, but this makes me a little
> uncomfortable.
>
> This could all be solved by including a th_decode_rgb_out() function.
> I realize this would be a convenience function, but if there's ever
> been a clearer need for convenience, I certainly can't think of one!
>
> 2. There is no example code to linearly interpolate the UV up to the
> Y.  I found on page 29 of the Theora Specification at http://theora.org/doc/Theora.pdf
>  that UV are centered on the 2x2 blocks of the Y.  I can come up with
> something that matches my conception of what's happening, but I worry
> that while videos are being encoded, that there is a standard for
> pixel centered or corner centered, which might end up with 1/2 pixel
> alignment errors.  I think that player_example.c should have a code
> snippet that shows how to properly align the channels.
>
> 3. player_example.c doesn't describe how to rewind the various state
> variables back to the beginning of the movie.  Seeking is probably
> overkill, but it's important to be able to rewind so that games can
> use movies that loop.
>
> 4. Timing is not well documented.  It took me a while to realize that
> th_granule_time() actually returns the granule time PLUS the duration
> of the current frame, returning the stale time, which is documented at http://www.theora.org/doc/libtheora-1.0/group__basefuncs.html#g707e1e281de788af0df39ef00f3fb432
>  but also needs a comment on the line that says
> "videobuf_time=th_granule_time(td,videobuf_granulepos);".  Also, there
> is no documentation that I can find that describes how Vorbis does
> timing, so I came up with the following code, where I calculate my own
> timing based on the current byte position in the audio stream:
>
> *audioStartTime = audiobuf_time;        // return the buffer time from the
> last call, which was the start of the buffer
> audiobuf_time = ((double) audiobuf_granulepos)/vi.rate;
> audiobuf_granulepos += (*numBytes/2)/2; // a granule is a left/right
> stereo pair, so the [size in bytes divided by 16 bits (2 bytes)
> divided by stereo (2 chans)]
> *audioStaleTime = audiobuf_time;
>
> As far as I can tell, this results in proper syncing but I could be
> off a frame and not know it...
>
> 5. player_example.c is much too complicated.  Now that my code is
> finished, this is my draw loop.  Under the hood, I'm using an RGBA
> texture in OpenGL and a buffered sound channel class a made on top of
> OpenAL.  I believe the code largely speaks for itself, but included
> comments on the confusing parts:
>
> ////////////////////////////////////////
>
> Video                   video;          // my wrapper class for Ogg Theora
> ImageBuffer             videoBuffer;    // my wrapper class for OpenGL texture
> BufferSoundChannel      bufChan;        // my wrapper class for OpenAL
>
> if( video.Load( "~/Desktop/ogg-theora-tests/320x240.ogg" ) != noErr )
> ErrorDialog( "Couldn't load file" );
>
> double          startTime = video.HasAudio() ? 0 : bufChan.GetTime();
>
> double          audioStartTime = 0, audioStaleTime = 0,
>                        videoStartTime = 0, videoStaleTime = 0;
> int                     audioStatus = kVideoAudioBuffering, videoStatus =
> kVideoVideoBuffering;
> Boolean         audioPlaying = false;
>
> while( !KeyDown( escape_key ) )
> {
>        short   *buffer;
>        uint    size;
>        double  theTime;
>
>        //if( video.Done() ) video.Reset();     // I don't have rewind working yet
>
>        video.Idle();   // buffer data even if no audio or video frames to
> display
>
>        theTime = bufChan.GetTime() - startTime;
>
>        if( audioStatus < 0 && bufChan.IsReady( (void**) &buffer,
> &size ) )       // see if the channel is ready to accept more audio, and if
> so, get the next temporary buffer in which to write the samples
>                audioStatus = video.GetAudio( buffer, &size, &audioStartTime,
> &audioStaleTime );      // returns the frame number of the audio, or a
> negative value to indicate wrong state or an error
>
>        if( audioStatus >= 0 )
>        {
>                theTime = bufChan.GetTime() - startTime;
>
>                if( theTime >= audioStartTime )
>                {
>                        if( !startTime ) startTime = bufChan.GetTime(); // start the sync
> at the exact moment we play the first buffer
>
>                        audioPlaying = true;
>
>                        bufChan.Play( buffer, size );   // queue the next audio buffer
>
>                        audioStatus = kVideoAudioBuffering;     // indicate that we are ready
> to read the next audio frame
>                }
>        }
>
>        theTime = bufChan.GetTime() - startTime;
>
>        if( videoStatus < 0 && (audioPlaying || !video.HasAudio()) )    // wait
> for the first audio buffer to play, because video can play faster to
> catch up, but audio is locked into the sample rate
>                videoStatus = video.GetVideo( videoBuffer, &viewRect,
> &videoStartTime, &videoStaleTime );     // returns the frame number of the
> video, or a negative value to indicate wrong state or an error
>
>        if( videoStatus >= 0 )
>        {
>                theTime = bufChan.GetTime() - startTime;
>
>                if( theTime >= videoStartTime )
>                {
>                        FlushToScreen();        // can just flush the texture buffer to the
> screen, because the video frame was written to videoBuffer when the
> frame was ready
>
>                        videoStatus = kVideoVideoBuffering;     // indicate that we are ready
> to read the next video frame
>                }
>        }
> }
>
> ////////////////////////////////////////
>
> Conceptually, the important parts are that I can grab the next audio
> or frame whenever I need them, and then I just wait until my buffered
> sound channel's timestamp reaches the audio and video buffers' start
> times.  Then I either queue the audio buffer samples or flush the
> video buffer to the screen.  I'm not calibrating my audio clock, but
> that's ok for the short duration videos we plan to use for now.
>
> IMHO, this is the example that people are looking for, and I don't see
> Theora gaining popularity in games as quickly if there is no high-
> level usage example like this.  I don't mean to sound harsh, it's just
> that I have a 4 year engineering degree and 20 years of programming
> experience and still found player_example.c to be quite cryptic.
> Conceptually it's very well written, and I appreciate things like
> select on the audio stream, but unfortunately with Mac and Windows
> operating systems, a lot of programmers today don't understand the
> simplicity of streams or even how to access them through the OS.
>
> Sometimes we take our experience for granted and expect others to
> either know what we are talking about or to read the entire manual and
> understand the fundamentals before using a library.  But in the real
> world, it's almost impossible for that to happen when there are
> divisions of labor and constant deadlines.
>
> If I can get the RGB gamma stuff working properly, I might consider
> releasing this code as an alternate player example that uses OpenGL
> and OpenAL.  It might take me a while, because my OpenGL and OpenAL
> stuff is tied pretty solidly into our engine, but maybe I could rig a
> barebones example with GLUT.  But I would like to see it happen :)
>
> Well I guess that's it, sorry for the long email, but I think these
> are issues that everyone is going to hit, so they need to be remedied.
>
> Thanxs for writing Ogg Theora, I think it's a wonderful library, just
> needs some better documentation and examples.  I tend to work on the
> back end of things and realize that getting someone to write concise
> example code is like herding cats.
>
> Zack Morris
> [email protected]
> _______________________________________________
> theora-dev mailing list
> [email protected]
> http://lists.xiph.org/mailman/listinfo/theora-dev
>