Re: darcs push hangs

Ben Franksen wrote:
> Ben Franksen wrote:
>> Manoj Gudi wrote:
>>> I am trying to commit a new log file (size 20Mb), `darcs record` works
>>> fine, however `darcs push` hangs after asking for credentials..
>>> 
>>> Any ideas?
>> 
>> You hit a long-standing weak spot of Darcs: transmitting large files can
>> take a long time.
> 
> tl;dr What we need to do in Darcs is to compress the bundle before sending
> it over the line. That's what git seems to do and it cuts down quite a bit
> on the transfer time.

Ok, here is my result:

darcs push -a --debug --verbose  24,59s user 1,24s system 23% cpu 1:50,67 
total

This is an improvement of more than factor 3 over the 6 plus minutes we 
needed before. And it was easy; this is all I had to do:

  * compress bundle before sending
    hunk ./src/Darcs/UI/External.hs 81
    -import Darcs.Util.ByteString (linesPS, unlinesPS)
    +import Darcs.Util.ByteString (linesPS, unlinesPS, gzWriteHandle)
    hunk ./src/Darcs/UI/External.hs 100
    +                 renderPSs,
    hunk ./src/Darcs/UI/External.hs 150
    -       hPutDoc target i inp
    +       gzWriteHandle i $ renderPSs target inp
    hunk ./src/Darcs/Util/ByteString.hs 32
    +        gzWriteHandle,
    hunk ./src/Darcs/Util/ByteString.hs 351
    +gzWriteHandle :: Handle -> [B.ByteString] -> IO ()
    +gzWriteHandle h pss  =
    +    BL.hPut h $ GZ.compress $ BL.fromChunks pss
    +

(Note: on the receiving side, Darcs already checks the input stream looking 
for a certain byte sequence that characterize gzip'ed data and automatically 
unzips in this case. I did *not* patch the darcs on the receiving side, so 
this change is compatible at least with the current HEAD. Will do a few more 
tests with 2.8.4/5 to make sure this works, too.)

I will send this a darcs patch (with a somewhat extended description).

Now, the time is still a bit more than double of what git (or gzip + ssh + 
gunzip) takes (remember that was a bit less than 50 seconds). To further cut 
down on the overhead would take a bit more effort.

I was, in fact, astonished to find that what Darcs does is to first create a 
normal patch bundle (in text form, just as when you 'darcs send' it), then 
transfers that, and on the receiving side applies it as it would apply any 
normal bundle.

What I think we should aim for is to decouple the 'meta data' (meaning, 
here, what the receiving side needs to know about which patches to apply, no 
compression needed for that) from the raw patch data. The latter we already 
have in compressed form (normally, i.e. unless you do 'darcs optimize 
uncompress'), so we could send the patch files as they are without any 
further processing. With this we should be able to be in the same league as 
git for large files or patches.

Cheers
Ben
-- 
"Make it so they have to reboot after every typo." -- Scott Adams