Re: Modifying "msg" and "file" files

Hello Nagy:

In message <20201125204545.74bedbe3@Dell>,
Nagy Gabor writes:
>I am new to roundup, and it seems very impressive;

Welcome to roundup.

>I really like its
>customizability. I bring up the same question as I asked on IRC today
>(sorry for the duplication, I think this is the better place for the
>discussion).

Yup. No problem on the duplication, but yes this is a better place for
the discussion.

>My question is that if I modify an existing "msg1" or "file1" file in
>the db directory of the tracker, then the Roundup database will remain
>consistent? (Or does the database rely on some properties of the
>created file, like size or md5sum etc.?)

By default, there is no validation of the file contents stored in the
db. The mime type used for the file is stored in the db. You might
want to change (e.g. to text/plain from application/octet-stream or
image/jpg) so people can view the edited file rather than being forced
to download it.

You can edit the file any way you like. If you delete the file,
retrieving it will cause an error. Emptying the file is also confusing
to the user. I recommend something like:

  echo "File deleted due to spam" > db/files/msg/0/msg344

so the user who tries to download the file will not be presented with
an empty download and think something has gone wrong.

>I want to do this, because sometimes it would be nice to edit
>messages or wipe out large files (e.g. email attachments) on the disk
>by replacing them to an empty files (which does not break the
>"retirement" design of roundup).

Yup I have edited messages to remove passwords 8-(.

Note retiring the file only removes it from indexes. You can still
download it if you specify the url e.g. tracker/msg334. Retiring the
item doesn't cause a 404 error or prevent accessing the info.

>I did not find the answer to this question in the documentation.

Yeah the docs need some restructuring. Where did you expect to find
that info?

>By inspection, it *seems* that modifying these files does not break
>anything, and I will give a deeper look to the relevant parts of the
>code. But I would be secure, if any of the main developers could confirm
>my "observation" / "conjecture", because probably I would never
>understand the whole source code like them. 

You can make these changes. It's not explicitly permitted by roundup
(i.e. roundup doesn't provide an edit mechanism for messages by
default). So while roundup expects the messages to be immutable, it
doesn't require/enforce that. Also file content changes are not
journaled.

You can use file mixins as documented at:

  https://wiki.roundup-tracker.org/MixinClassFileClass

to do various things with the stored files. The two examples there
de-duplicate attached files and perform transparent file
compression.

You could add a content_md5sum field to the schema. Then write a mixin
that stores an md5 sum of the file contents in the database. The mixin
could validate the value on retrieval and report a problem if the
validation fails. The mixin could also add another journal entry
reporting that content changed. The journal entries would be shown
providing some indication that the content was changed.

>And if the modification of these files are (unofficially) supported,
>then what is your suggested method for doing that (I mean, using as much
>Roundup API as possible).

You can do it via rest (assuming you are using round 2.0). For example:

   $ curl -u demo:demo -s -H "X-requested-with: rest" \
                          -H "Referer: https://rouilj.example.com/demo/" \
                          -X GET \
             https://rouilj.example.com/demo/rest/data/file/30/content
  {
      "data": {
          "id": "30",
          "type": "<class 'str'>",
          "link": "https://rouilj.dynamic-dns.net/demo/rest/data/file/30/content",
          "data": "hello3",
          "@etag": "\"3f2f8063dbce5b6bd43567e6f4f3c671\""
      }
  }

Save the etag and use it to overwrite the content:

   $ curl -u demo:demo -s -H "X-requested-with: rest" \
                          -H "Referer: https://rouilj.example.com/demo/" \
                          -H 'If-Match: "3f2f8063dbce5b6bd43567e6f4f3c671"' \
                          -X PUT \
                          -F "data=@hello" \
                https://rouilj.dynamic-dns.net/demo/rest/data/file/30/content

where hello is a file on local disk.

Also since roundup usually considers the file immutable, you can just
edit the file on disk as well. (You can create an auditor to prevent
updating the content property to enforce immutability.)

Hopefully this helps.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.