Re: move prebuilt libs from svn to git?

Congratulations Dan!  I know this has been something that's been 
problematic for a long time, so it's great you were able to improve 
things (and by the sound of it, things have improved by a lot)

Ryan Inch (Imaginer)

On 2022-06-17 17:45, Dan McGrath via Bf-committers wrote:
> Just a heads-up that Ray and I took a packet capture on both ends for me.
>
> Thanks to his capture, I noticed that the SVN server host was not sending
> TCP window scale or SACK options. Ultimately this turned out to be due to
> that host using the PF firewall "synproxy" state option, which apparently
> was causing the TCP header options to be removed, which limited the window
> size to 64k.
>
> I have corrected this change to match the www host, and we did observe some
> improvements that were hidden from me before due to my and the studio IPs
> special treatment that bypassed this feature.
>
> While it should help, I also tuned a few sysctl settings for send and
> receive buffers. In addition, I noticed that the SVN server is also not
> sending gzip/deflate accept headers for the files. No doubt this is going
> to be a factor. However, there was a note from 2016 from myself explaining
> that Brecht asked for the SVN compression to be disabled due to some client
> problems. So, this will require some research, perhaps next week.
>
> Anyway, hopefully it helps a bit until I can review some more settings.
>
> Enjoy your weekend everyone!
>
>
> Dan
>
> On Fri, Jun 17, 2022, 2:22 PM Ray Molenkamp via Bf-committers <
> [email protected]> wrote:
>
>> =8<==[Editors note]=8<==
>> I'll be honest I'm 100% convinced mailinator will ruin
>> the little formatting I have done, I put a copy of this
>> email on https://developer.blender.org/P3014.
>> =8<====8<====8<====8<==
>>
>> How much as I like the sentiment of "lets move to git will it
>> solve all these problems" lets be honest here, git.blender.org's
>> speed is nothing to write home about either it may or may not be
>> as glitchy as svn, but it still wouldn't be fast.
>>
>> from my 300mbit connection in western Canada
>>
>> Receiving objects:   6% (135931/2078159), 33.34 MiB | 458.00 KiB/s
>>
>> total time for a clone off git.blender.org 27m39s
>>
>> Cloning off https://github.com/blender/blender.git however
>>
>> total time for a clone off github.com 1m 52s
>>
>> Now amount of finger pointing to client settings will convince me
>> it's a client setting at this point, but I'll be honest, I am all
>> the way in western Canada and that could definitely be a factor.
>>
>> sooooo, lets investigate!
>>
>> Let’s spin up an AWS EC2 instance in London, I'd say that be close
>> enough to Amsterdam, and let’s rule out any other CPU or Bandwidth
>> related factors as well, I threw a few dollars at it and got a
>> cn5.18xlarge instance (72 CPU’s 192gb memory, 100gbit ethernet)
>> overkill to do a git/svn test? yes.. yes indeed
>>
>> first to rule out the blender.org servers lets grab a ubuntu iso
>> off an .nl mirror (
>> https://mirror.nl.datapacket.com/ubuntu-releases/22.04/ )
>>
>> 2022-06-17 17:13:21 (232 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’ saved
>> [3654957056/3654957056]
>>
>> all right bandwidth is NOT going to be an issue on this box!
>>
>> On to blender stuff:
>>
>> cloning from git.blender.org  2m32.895s
>> svn checkout of lib/win64_vc15 2m56.388s (iftop said 65Mbit peak rate)
>>
>> yowza!
>>
>> All right, so in London, safe to say all is well
>>
>> lets move closer to my location, us-east-2 Ohio (same instance type and os)
>>
>> grabbing ubuntu from that same mirror
>>
>> 2022-06-17 17:31:00 (29.5 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’ saved
>> [3654957056/3654957056]
>>
>> all right, that has lost quite a bit of steam, but it's
>> still nothing to sneeze at, just to be sure lets grab
>> an iso off the Princeton university mirror 120MB/s ok..
>> good enough...
>>
>> onto cloning blender
>>
>> Receiving objects:   5% (104235/2078173), 25.44 MiB | 692.00 KiB/s
>>
>> well... that's not looking promising... lets wait it out
>>
>> cloning from git.blender.org  17m23.449s
>>
>> sadface... lets try svn next, i lost patience and did not let
>> it finish.. iftop reported a top RX of 5.59Mbit though..
>>
>> To summarize:
>>
>> Ubuntu iso from mirror.nl.datapacket.com (Speed taken from wget)
>> My home - western Canada - 1.3  MB/s
>> AWS     - London         - 232  MB/S
>> AWS     - Ohio           - 29.5 MB/s
>>
>> Clone of git.blender.org (time taken from linux time command)
>> My home - western Canada - 27m39
>> AWS     - London         -  2m32
>> AWS     - Ohio           - 17m23
>>
>> SVN Clone: (peak RX bitrate taken from iftop)
>> My home - western Canada -  1.5 Mbit/s
>> AWS     - London         -   64 Mbit/s
>> AWS     - Ohio           - 5.59 Mbit/s
>>
>> Think the only thing we really can conclude is
>> that being further from the server is leading
>> to an "unhappy" time for the developer. Given
>> the fact the EC2 instances were 100% identical
>> between London and Ohio, it's unlikely to be a
>> client configuration issue.
>>
>> --Ray
>>
>> On 2022-06-17 10:10 a.m., Dan McGrath via Bf-committers wrote:
>>> Hi,
>>>
>>> Really, I would need to see a capture from both sides to dig into it. I
>>> need to see what is being sent, and what is received on each side. There
>>> could be some weird firewall stuff happening that is causing
>> fragmentation
>>> or blocking TCP options, or window scaling issues, etc.
>>>
>>> As for the default settings for tortoisesvn, the connection stuff I was
>>> referring to was about TCP/IP behaviour, rather than an application
>>> setting. Either way, I need to see what is actually happening. The fact
>>> that I can pull 5-8MB/sec (~40mbit) from Canada and not have a single
>>> interruption is telling. Granted, the host only has around 100 or 200mbit
>>> allocated, iirc. Multiply that by a few dozen people and you have a
>> problem.
>>> For a simpler test, maybe try a large svn check, perhaps some old FreeBSD
>>> ports svn repo if you can find one, something also in .NL, and see how it
>>> goes, just to help eliminate your PC/firewall. Worst case scenario, you
>> can
>>> also poke me in chat and we can try some server side and client side
>>> captures.
>>>
>>> TBH though, 1gig uplink just isn't a whole lot to serve our user base ;)
>>>
>>> On Fri, Jun 17, 2022 at 11:45 AM Tom M <[email protected]> wrote:
>>>
>>>> On Fri, Jun 17, 2022 at 5:20 AM Dan McGrath <[email protected]>
>>>> wrote:
>>>>> Hi Tom,
>>>>>
>>>>> Long time no see :)
>>>>>
>>>>> I did some tests from home and found that, aside from slow speeds, I
>> was
>>>> able to checkout at least 5 or 6 gig of the bf-blender repo without
>> error
>>>> or having to retransmit.
>>>>
>>>> Yeah, looks like I was mistaking really slow download of the larger
>>>> (.7 GB) libraries as freezing.  I was averaging about .3 MB/s.  I used
>>>> a different SVN client and saw that it was indeed downloading (the
>>>> sliksvn doesn't provide any feedback on files in progress, only once a
>>>> file completes) - It was taking about 40 minutes for each of the
>>>> larger files.  It did complete eventually after about 5 hours.
>>>>
>>>>> I took the liberty of packet capturing the stream from my PC's POV, and
>>>> placed them in the ftp download area[1], if you care to take a look.
>>>>> The gist is that I was able to get around 5MB/sec for at least 5G of
>> the
>>>> checkout, before I cancelled it.
>>>> Interesting, I usually get up to 3 MB/s download here normally from
>>>> other sources, but was typically getting .3 MB/s, with occasional
>>>> bursts of higher speed to a max of .8 MB/s when it wasn't downloading
>>>> the large files.
>>>>
>>>>> Now, my IP is a bit of a special case, so I am wondering if you are
>> just
>>>> hitting the server so hard with http requests in separate sockets, or
>>>> something extreme, that is causing the system to block you for a period
>> of
>>>> time. For comparison, my PC only had about 4 sockets open to the server
>> for
>>>> the transfer. I would be curious to know if you are doing pipelining, or
>>>> sending each request in a separate connection.
>>>> Whatever the sliksvn and tortoisesvn defaults are, I'd assume they are
>>>> setup for typical usage.
>>>>
>>>>> If you can, the next time it happens, see if you are unable to visit
>>>> Phabricator or resume the svn checkout process. It may not be a firewall
>>>> issue, but without seeing some packet captures, I wouldn't be able to
>>>> really guess. Ideally, try to capture a few packets, until failure. At
>> the
>>>> very least, you will need to snap 20b+20b=40b for IP+TCP headers, but I
>>>> would suggest a tad higher, maybe 64b.
>>>>> Cheers,
>>>>>
>>>>> Dan
>>>>>
>>>>> [1] - https://download.blender.org/ftp/dan/svn-checkout/
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 16, 2022 at 2:43 PM Tom M via Bf-committers <
>>>> [email protected]> wrote:
>>>>>> When checking out the libraries from SVN following the directions on
>>>>>>
>>>>>> https://wiki.blender.org/wiki/Building_Blender/Windows
>>>>>>
>>>>>> The svn checkout of the libraries fails with significant frequency,
>>>>>>
>>>>>> if you google you might stumble across using make svnfix
>>>>>>
>>>>>>
>> https://www.mail-archive.com/[email protected]/msg151996.html
>>>>>> Even after using it, I'm still having to do Ctrl C, and then make
>>>>>> svnfix repeatedly (it downloads some files, than randomly freezes,
>>>>>> sometimes after 20-50 files, sometimes after 100's of files).
>>>>>>
>>>>>> Git doesn't seem to have any problems, though.
>>>>>>
>>>>>> So might we consider migrating the libs to git, rather than using SVN?
>>>>>>
>>>>>> LetterRip
>>>>>> _______________________________________________
>>>>>> Bf-committers mailing list
>>>>>> [email protected]
>>>>>> List details, subscription details or unsubscribe:
>>>>>> https://lists.blender.org/mailman/listinfo/bf-committers
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Danny
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Danny McGrath - [email protected]
>>>>> GPG key: EDF6 AFF5 2086 F93A 1F59 36A5 44B6 26F3 6968 71CA
>> _______________________________________________
>> Bf-committers mailing list
>> [email protected]
>> List details, subscription details or unsubscribe:
>> https://lists.blender.org/mailman/listinfo/bf-committers
>>
> _______________________________________________
> Bf-committers mailing list
> [email protected]
> List details, subscription details or unsubscribe:
> https://lists.blender.org/mailman/listinfo/bf-committers

_______________________________________________
Bf-committers mailing list
[email protected]
List details, subscription details or unsubscribe:
https://lists.blender.org/mailman/listinfo/bf-committers