A proposal to create a kind of 'delta metadata' (for XML md for now)

Hedayat Vatankhah <[email protected]> Wed, 30 Jul 2014 01:15:15 +0430
Newsgroups gmane.linux.rpm.metadata
Message-ID <[email protected]> 
This is a multi-part message in MIME format.
--===============4074671598563664516==
Content-Type: multipart/alternative;
 boundary="------------030009070109030903030801"

This is a multi-part message in MIME format.
--------------030009070109030903030801
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Dear all,
While I'm thinking about a completely new repository format, but I found 
that would need much time from my side and probably lots of discussions. 
But I think that implementing a kind of delta metadata support for the 
current format is much easier and possibly provides considerable 
improvement for updates compared to the current situation where a 
completely new metadata is generated for updates. And I think I can 
contribute code too.

Anyway, I'd like to propose my idea and request for feedback. My 
suggestion is currently limited to XML MD format only, which is more 
space efficient than SQLite dbs in itself. And specially since DNF is 
using them.

The idea is very simple: when createrepo is going to update an existing 
repodata, it doesn't replace the old primary/filelists/etc data. 
Instead, it creates new xml files containing data about 
added/modified/deleted packages (a deleted package can be marked by 
having an entry for it without any data except its name and EVR. A 
special attribute might be added if needed). And, repomd.xml contains 
reference to these new files only.

Now, how older metadata files are referenced? They are referenced from 
the new MD files, just like how repomd.xml refers to the latest 
versions. For example, new primary.xml file which contains data about 
latest added/modified/deleted packages references the previous 
primary.xml file (which in turn might reference an older primary.xml 
database). Therefore, we have a linked list of primary.xml files, where 
the head is the latest primary delta and the tail is the main 
primary.xml file created when the repo was created.

I like the idea of linked lists here, because it is scalable! But if it 
seems overkill (does not allow pipelining to download a number of delta 
xml files, because you should download an open an XML file to learn 
about the name of previous one), we might come up with some other 
suggestions for storing the list, e.g. the list might be stored in 
repomd.xml itself, or a separate file for interested clients.

IMHO, this can be easily implemented in createrepo, and should be fairly 
straightforward for clients to use. What do you think?

Regards,
Hedayat

--------------030009070109030903030801
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit

<html style="direction: ltr;">
  <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
    <style type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>
  </head>
  <body style="direction: ltr;"
    bidimailui-detected-decoding-type="UTF-8"
    smarttemplateinserted="true" bgcolor="#FFFFFF" text="#000000">
    Dear all,<br>
    While I'm thinking about a completely new repository format, but I
    found that would need much time from my side and probably lots of
    discussions. But I think that implementing a kind of delta metadata
    support for the current format is much easier and possibly provides
    considerable improvement for updates compared to the current
    situation where a completely new metadata is generated for updates.
    And I think I can contribute code too. <br>
    <br>
    Anyway, I'd like to propose my idea and request for feedback. My
    suggestion is currently limited to XML MD format only, which is more
    space efficient than SQLite dbs in itself. And specially since DNF
    is using them.<br>
    <br>
    The idea is very simple: when createrepo is going to update an
    existing repodata, it doesn't replace the old primary/filelists/etc
    data. Instead, it creates new xml files containing data about
    added/modified/deleted packages (a deleted package can be marked by
    having an entry for it without any data except its name and EVR. A
    special attribute might be added if needed). And, repomd.xml
    contains reference to these new files only. <br>
    <br>
    Now, how older metadata files are referenced? They are referenced
    from the new MD files, just like how repomd.xml refers to the latest
    versions. For example, new primary.xml file which contains data
    about latest added/modified/deleted packages references the previous
    primary.xml file (which in turn might reference an older primary.xml
    database). Therefore, we have a linked list of primary.xml files,
    where the head is the latest primary delta and the tail is the main
    primary.xml file created when the repo was created. <br>
    <br>
    I like the idea of linked lists here, because it is scalable! But if
    it seems overkill (does not allow pipelining to download a number of
    delta xml files, because you should download an open an XML file to
    learn about the name of previous one), we might come up with some
    other suggestions for storing the list, e.g. the list might be
    stored in repomd.xml itself, or a separate file for interested
    clients. <br>
    <br>
    IMHO, this can be easily implemented in createrepo, and should be
    fairly straightforward for clients to use. What do you think?<br>
    <br>
    Regards,<br>
    Hedayat<br>
  </body>
</html>

--------------030009070109030903030801--

--===============4074671598563664516==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Rpm-metadata mailing list
[email protected]
http://lists.baseurl.org/mailman/listinfo/rpm-metadata

--===============4074671598563664516==--