Re: storage engine: understanding the row format

Hi =C3=98ystein,

thanks for the fast and helpful reply!

I assume that table->s->null_bytes tells me how many bytes are used to
describe the nullable columns?

Best regards
Christoph

2016-05-19 14:34 GMT+02:00 =C3=98ystein Gr=C3=B8vlen <oystein.grovlen@oracl=
e.com>:
> Hi Christoph,
>
> On 19. mai 2016 14:15, Christoph Rupp wrote:
>>
>> Hi,
>>
>> If a row has variable-length blobs (fields of type
>> MYSQL_TYPE_VARCHAR), then the serialized row stores the full length of
>> the blob, even if most bytes are unused. In such cases, i'd like to
>> "compress" the row before writing it to disk.
>
>
> Note that VARCHAR and BLOB are different types.  Contrary to VARCHAR, for
> blobs, space is not allocated in the internal record buffer, but in separ=
ate
> buffers.
>
>>
>> However, i have a few difficulties understanding the row format. The
>> following two CREATE TABLE statements are relatively similar (the
>> first one creates an additional index). But their first byte differs,
>> and I don't understand why.
>
>
> The index is not the only difference.  Nullability of the value column al=
so
> differs.
>
>>
>> CREATE TABLE test (value VARCHAR(30) NOT NULL, INDEX(value), num
>> INTEGER PRIMARY KEY)
>> INSERT INTO test VALUES("1", 1);
>>
>> (gdb) x/8b buf
>> 0x7fff5000e660:    1    49    -113    -113    -113    -113    -113    -1=
13
>>
>> buf[0] stores the length of 'value', buf[1] stores the data of 'value'.
>>
>> CREATE TABLE test (value VARCHAR(30), num INTEGER PRIMARY KEY)
>> INSERT INTO test VALUES("1", 1);
>>
>> (gdb) x/8b buf
>> 0x7fff50013cf0:    -2    1    49    -113    -113    -113    -113    -113
>>
>> Now buf[1] stores the length and buf[2] stores the data of 'value'.
>>
>> But what is buf[0]?
>
>
> The additional byte(s) is used to record which nullable columns are NULL.
> For tables where there are no nullable columns, there will not be such a
> byte.
>
> Regards,
>
> --
> =C3=98ystein
>
>
>>
>> Is there documentation for the serialized row format?
>>
>> Thanks
>> Christoph
>>
>> PS: here's the code that i currently use:
>>
>> static inline ups_record_t
>> pack_record(TABLE *table, uint8_t *buf, uint8_t *arena)
>> {
>>    assert(!row_is_fixed_length(table));
>>
>>    uint8_t *src =3D buf;
>>    uint8_t *dst =3D arena;
>>
>>    // copy the first byte - whatever it is
>>    // this causes problems because in some cases there is no "first byte=
"!
>>    *dst =3D *src;
>>    dst++;
>>    src++;
>>
>>    for (Field **field =3D table->field; *field !=3D 0; field++) {
>>      uint32_t type =3D (*field)->type();
>>      uint16_t key_size;
>>      uint32_t len_bytes;
>>
>>      if (type =3D=3D MYSQL_TYPE_VARCHAR) {
>>        // see Field_blob::Field_blob() (in field.h) - need 1-4 bytes to
>>        // store the real size
>>        if ((*field)->field_length <=3D 255) {
>>          len_bytes =3D 1;
>>          key_size =3D *src;
>>        }
>>        else if ((*field)->field_length <=3D 65535) {
>>          len_bytes =3D 2;
>>          key_size =3D *(uint16_t *)src;
>>        }
>>        else if ((*field)->field_length <=3D 16777215) {
>>          len_bytes =3D 3;
>>          key_size =3D *src; // TODO implement this
>>        }
>>        else {
>>          len_bytes =3D 4;
>>          key_size =3D *(uint32_t *)src;
>>        }
>>      }
>>      else {
>>        len_bytes =3D 0;
>>        key_size =3D (*field)->key_length();
>>      }
>>
>>      ::memcpy(dst, src, key_size + len_bytes);
>>      src +=3D (*field)->pack_length();
>>      dst +=3D key_size + len_bytes;
>>    }
>>
>>    ups_record_t r =3D ups_make_record(arena, (uint32_t)(dst - arena));
>>    return r;
>> }
>>
>
>
> --
> =C3=98ystein

-- 
MySQL Internals Mailing List
For list archives: http://lists.mysql.com/internals
To unsubscribe:    http://lists.mysql.com/internals