[Offtopic]| half lossy datatype

Gonzalo Garramuno <[email protected]>
Newsgroups gmane.comp.video.openexr.devel
Message-ID <[email protected]>
I find myself with updating ffmpeg's exr reader (don't ask). One thing i 
am trying to do is add a gamma function to the reading of the format, 
with a gamma table.
However, I find myself against a wall not being able to get the float 
and half float to behave correctly.  I end up with greens in places 
where there were negative values or nans.
I am wondering if someone could take a look at these functions I am 
using to see if some bit is being set wrong.  Or if it is indeed 
impossible to have half->float->half (as done in my filling of the 
table) mantaining the same values (with gamma 1.0).

/**
  * Convert from 32-bit float as uint32_t to uint16_t.
  *
  * @param v 32-bit float
  *
  * @return normalized 16-bit unsigned int
  */
static inline uint16_t exr_flt2uint(uint32_t v)
{
     unsigned int exp = v >> 23;
     // "HACK": negative values result in exp<  0, so clipping them to 0
     // is also handled by this condition, avoids explicit check for 
sign bit.
     if (exp <= 127 + 7 - 24) // we would shift out all bits anyway
         return 0;
     if (exp >= 127)
         return 0xffff;
     v &= 0x007fffff;
     return (v + (1 << 23)) >> (127 + 7 - exp);
}


/**
  * Convert from 16-bit float as uint16_t to uint16_t.
  *
  * @param v 16-bit float
  *
  * @return normalized 16-bit unsigned int
  */
static inline uint16_t exr_halflt2uint(uint16_t v)
{
     unsigned exp = 14 - (v >> 10);
     if (exp >= 14) {
         if (exp == 14)
             return (v >> 9) & 1;
         else
             return (v & 0x8000) ? 0 : 0xffff;
     }
     v <<= 6;
     return (v + (1 << 16)) >> (exp + 1);
}

/*
  * Convert a half float as a uint16_t into a full float
  *
  */
static inline float exr_half2float(uint16_t h)
{
     static const union av_intfloat32 magic = { (254 - 15) << 23 };
     static const union av_intfloat32 was_infnan = { (127 + 16) << 23 };
     union  av_intfloat32 o;

     o.i = (h & 0x7fff) << 13; /* exponent/mantissa bits */
     o.f *= magic.f; /* exponent adjust */
     if (o.f >= was_infnan.f) /* make sure Inf/NaN survive */
         o.i |= 255 << 23;
     o.i |= (h & 0x8000) << 16; /* sign bit */

     return o.f;
}


For the gamma table, I am doing:

     float one_gamma = 1.0f / gamma;
     for ( uint32_t i = 0; i < 65536; ++i ) {
        t.f = exr_half2float(i);
        t.f = powf(t.f,one_gamma);
        s->gamma_table[i] = exr_flt2uint(t.i);
     }
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.