Re: dar encryption - RSA

Hi Denis,

thank you very much for the official feature request. I know it takes
time to implement such a complex feature and you have other open points
too. If I can help you in some point please let me know.

About the hashes, one salt will be used for one archive.
An attacker shouldn't be able to use one hash twice (on two different
archives), this is the purpose of the salt. The salt does not make the
hash more secure in general, because it is public. It has to be stored
beside the hash table outside the encrypted area of the archive.
The only attack I can think about is the situation when the attacker can
guess the path and file name. Then he could proof that one specific file
is in the archive. To prevent this case I added the other values:
H(path+filename + inodeID + mtime + UUID + salt)
It is meant that only the inodeID + mtime + UUID provide enough
security to make it inefficient to crack the hash.
The time an attacker would need to do so can be calculated as follows.
We assume:
* the path and file name can be guessed
* the attacker don't have to try more then 100000 inodeIDs
* mtime can be limited within one day
* a 32Bit UUID (meaning 32Bits that can't be guessed)
* 1 hash round (if it doesn't bother you we can go up to 100 rounds)
* a GPU cluster (http://hashcat.net/oclhashcat/#performance)
The possibilities that have to be brute-forced are:
100000*60*60*24*2^32
When we assume the attacker can generate 2005M c/s he will need:
((100000*2^32*60*60*24) / (2005*10^6))[s]
~ 580 years
I guess this is quite to long to wait.


I have created an demo application for the hashed catalogue:
https://github.com/peckto/hash_dic_test
It creates the hashes as discussed (1*SHA3-512) and stores them plus
the corresponding data in an std::unordered_map. Afterwards it iterates
again though the file system and looks up every hash in the map.
It turns out that the most time consuming part is the hashing process.
On my notebook it looks like this:
# ./hash_dic_test /
generate hashes
duration: 0:6:986
---------------------------------
build hash table
duration: 0:7:544
---------------------------------
search in hash table
cant find hash!
/var/log/journal/ad1d17f14aee4b34a7e6c6a3689ac394/system.journal|655806|1413144061|fdf7c30d-838c-4e16-af37-2a345650590a
duration: 0:7:563
---------------------------------
map entries: 495.425
map size: ~99,085MB

You can see that files that have been changed since the hash table has
been created, are not found in the table. 
About the performance, it takes about 7s to iterate though the file
system and to calculate the hashes. In the second stage the same process
is done again but the hashes and the related date is stored in the map.
In the last stage the hashes are generated again and are looked up in
the map. In reality the hash must only be calculated once to perform the
two different tasks (write the hash to the new table and look-up in the
table of reference). Both tasks would need about one second in total
(for 500000 files). You can try it on your own system as well. Du you
have any concerns regarding the performance?


How flexible is you archive, is it possible to get just some space (eg.
100MB) to store any sort of binary data? Are there any limitations?
I'm still working on a method to store the dictionary inside another
file (the dar archive). I'm looking at Berkeley db, it could also
replace the whole map structure.
It is also possible to just serialize the map (eg with boost), but this
is not very compatible.

Regards,
Tobias


Am Dienstag, den 07.10.2014, 21:26 +0200 schrieb Denis Corbin:
> Le 07/10/2014 17:41, Tobias Specht wrote:
> > Hi Denis,
> 
> Hi Tobias,
> 
> > 
> > maybe I was not that exactly about what I want to hash and how the 
> > dictionary is organized: * I don't want to hash the content of the
> > file
> This I understood,
> 
> > * the dictionary is not organized hierarchical as your catalogue
> > is
> this is didn't but OK, that does not change much the picture and makes
> sens to avoid exposing the directory tree structure.
> 
> > * when I'm talking about filename I mean path + file name
> OK,
> 
> > * the dictionary does not replace the catalogue, it is just an
> > extra option
> I understood that the dictionary was stored in clear text beside the
> catalogue which would stay encrypted.
> 
> 
> > It should look like this: { H("/home/tobias/Documents/test.txt" +
> > inodeID + mtime + UUID + salt) : [userID, groupID, perm, file_size,
> > is_dir, type, flags, ctime] ,
> > 
> > H("/home/tobias/Pictures/foo.jpg" + inodeID + mtime + UUID + salt)
> > : [userID, groupID, perm, file_size, is_dir, type, flags, ctime] , 
> > ... } (H() is a cryptographic hash function like sha256)
> > 
> > respectively: { 
> > b2144d23ebc9a7f2af44e215b00dce5025bdc227346c6459b989ef8d203f3402 : 
> > [userID, groupID, perm, file_size, is_dir, type, flags, ctime] ,
> > 
> > 0df9ba289c76d5bb1761a2764593bfe97d64f4c944ecfa08d6f7a16721b5f317 : 
> > [userID, groupID, perm, file_size, is_dir, type, flags, ctime] , }
> > 
> > 
> > In this scenario the only possibility for a collision to occur is
> > inside the hash function, which is very unlikely to happen: 
> > http://stackoverflow.com/questions/4014090/is-it-safe-to-ignore-the-possibility-of-sha-collisions-in-practice/4014407#4014407
> >
> > 
> => In my opinion the possibility of a hash collision can be ignored.
> 
> I admit the probability is very low, but this has to be documented at
> least for the user to know the risk, as low at it can be.
> 
> > 
> >> In fact, adding system/hardware ID in the hash forbids the
> >> possibility to restore the whole data (most probably on a new
> >> filesystem, due to a crash for example), and keep using the
> >> latest backup of reference as reference for the next incremental
> >> backup.
> > Yes, that's right. But this is not only because of the uuid it's
> > also because I want to use the inode number, which will be
> > different after the restore also.
> Yes, that's correct. I just wonder why adding the inodeID and UUID?
> Would just salt not be sufficient to randomize the data to hash? By
> the way, I suspect there would be a different salt value per hash?
> Would the salt for each entry be stored in clear beside the
> corresponding hash? No offense, my cryptographic knowledge is quite
> basic! :)
> 
> > In this case the user has to enter the encryption password to use
> > the encrypted catalogue as reference or a full backup will be
> > created. I think this restriction is acceptable.
> It is for me too. As you say, there is the catalogue for that situation.
> 
> > 
> > Of course I can use the same password for all backups of one system
> > and requesting the user only once to enter it (this can be done
> > without modifying dar, just by using libdar) but that's not the
> > point.
> > 
> > I admit the dictionary is not that easy to implement and it will
> > require changes on the archive format as well but I think it can be
> > quite handy for a lot of users who want to encrypt there backups.
> The archive format is flexible, so that's not a problem to add a new
> fields. The point concerns more the algorithme of differential backup
> (filtre.cpp: the filtre_sauvegarde() routine) It should be able to
> handle hashes in place of filename while also performing file
> comparison on filename (for normal differential backup).
> 
> Another point to consider is the algorithm complexity (I mean the time
> to execute the requested task). Actually when doing a differential
> backup, each file from the filesystem under backup first search in the
> reference catalogue, but only in the directory it is located in. Here,
> due to the hash on the whole path+filename, each new file to consider
> for backup has to be hashed and this hash has to be compared more
> widely to the whole archive hash base. Of course having a sorted list
> of hashes (like it is for filename in each directory) leads to a
> faster search (binary search) but it remains that the execution time
> will increase with number of files in the archive. I guess, this hash
> lookup is not the biggest CPU consuming task in libdar (comparing with
> data compression or encryption), but that's however a scalability issue.
> 
> I think I now get the picture of your request/idea. This is a
> reasonable compromise, while it is not a simple feature to
> implement... :-/
> 
> I add it to the Feature Request list on sourceforge.
> https://sourceforge.net/p/dar/feature-requests/173/
> 
> I can't promise I will have time to implement it for release 2.5.0,
> the next major release I would like to finish developping this year
> for a release first semester 2015. I'm taking more time than expected
> testing the current feature (multi-threaded libdar), while performance
> benefit is not much visible for now... well I have not yet tuned it
> all, first have to make it work as expected. So I can't promise but I
> will try to.
> 
> 
> > 
> > Regards, Tobias
> > 
> > 
> 
> Regards,
> Denis.



------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho