Re: move prebuilt libs from svn to git?
Ankit via Bf-committers <[email protected]> Sat, 18 Jun 2022 13:21:18 +0530
| Newsgroups | gmane.comp.video.blender.devel |
|---|---|
| Message-ID | <[email protected]> |
Zack - To track library versions https://developer.blender.org/T88438 - To tag corresponding versions for each release https://svn.blender.org/svnroot/bf-blender/tags/ - Some bugs show up due to deps also, https://developer.blender.org/rBL62677 . VCS can point to the commit that caused it. Ankit On 18/06/22 12:34, Zack Brown via Bf-committers wrote: > Hi everyone, > > It's fascinating to see this discussion. It never occurred to me that > anyone might continue to prefer svn once git came into existence, and yet > here is a sample case, right under my nose! I guess svn is still relevant > even after all these years! > > But actually I'd be curious to know why revision control is used at all in > this particular case. We're talking about precompiled binaries, right? > That's sort of the poster child for situations where revision control is > *not* what one wants, exactly for the reason why git would be a bad choice. > > Please, could someone explain why the blender project doesn't simply use a > dev-accessible shared folder somewhere? I love revision control, but it's > only useful if it's useful. > > Be well, > Zack > > > On Sat, Jun 18, 2022 at 1:10 AM Ray Molenkamp via Bf-committers < > [email protected]> wrote: > >> Just an update with some new numbers, dan's changes >> brought down the time for a full clone from my home >> connection down to 5m48s (down from 27m39s previously) >> >> well done dan! >> >> --Ray >> >> On 2022-06-17 3:45 p.m., Dan McGrath wrote: >>> Just a heads-up that Ray and I took a packet capture on both ends for me. >>> >>> Thanks to his capture, I noticed that the SVN server host was not >> sending TCP window scale or SACK options. Ultimately this turned out to be >> due to that host using the PF firewall "synproxy" state option, which >> apparently was causing the TCP header options to be removed, which limited >> the window size to 64k. >>> >>> I have corrected this change to match the www host, and we did observe >> some improvements that were hidden from me before due to my and the studio >> IPs special treatment that bypassed this feature. >>> >>> While it should help, I also tuned a few sysctl settings for send and >> receive buffers. In addition, I noticed that the SVN server is also not >> sending gzip/deflate accept headers for the files. No doubt this is going >> to be a factor. However, there was a note from 2016 from myself explaining >> that Brecht asked for the SVN compression to be disabled due to some client >> problems. So, this will require some research, perhaps next week. >>> >>> Anyway, hopefully it helps a bit until I can review some more settings. >>> >>> Enjoy your weekend everyone! >>> >>> >>> Dan >>> >>> On Fri, Jun 17, 2022, 2:22 PM Ray Molenkamp via Bf-committers < >> [email protected]> wrote: >>> >>> =8<==[Editors note]=8<== >>> I'll be honest I'm 100% convinced mailinator will ruin >>> the little formatting I have done, I put a copy of this >>> email on https://developer.blender.org/P3014. >>> =8<====8<====8<====8<== >>> >>> How much as I like the sentiment of "lets move to git will it >>> solve all these problems" lets be honest here, git.blender.org < >> http://git.blender.org>'s >>> speed is nothing to write home about either it may or may not be >>> as glitchy as svn, but it still wouldn't be fast. >>> >>> from my 300mbit connection in western Canada >>> >>> Receiving objects: 6% (135931/2078159), 33.34 MiB | 458.00 KiB/s >>> >>> total time for a clone off git.blender.org <http://git.blender.org> >> 27m39s >>> >>> Cloning off https://github.com/blender/blender.git however >>> >>> total time for a clone off github.com <http://github.com> 1m 52s >>> >>> Now amount of finger pointing to client settings will convince me >>> it's a client setting at this point, but I'll be honest, I am all >>> the way in western Canada and that could definitely be a factor. >>> >>> sooooo, lets investigate! >>> >>> Let’s spin up an AWS EC2 instance in London, I'd say that be close >>> enough to Amsterdam, and let’s rule out any other CPU or Bandwidth >>> related factors as well, I threw a few dollars at it and got a >>> cn5.18xlarge instance (72 CPU’s 192gb memory, 100gbit ethernet) >>> overkill to do a git/svn test? yes.. yes indeed >>> >>> first to rule out the blender.org <http://blender.org> servers lets >> grab a ubuntu iso >>> off an .nl mirror ( >> https://mirror.nl.datapacket.com/ubuntu-releases/22.04/ ) >>> >>> 2022-06-17 17:13:21 (232 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’ >> saved [3654957056/3654957056] >>> >>> all right bandwidth is NOT going to be an issue on this box! >>> >>> On to blender stuff: >>> >>> cloning from git.blender.org <http://git.blender.org> 2m32.895s >>> svn checkout of lib/win64_vc15 2m56.388s (iftop said 65Mbit peak >> rate) >>> >>> yowza! >>> >>> All right, so in London, safe to say all is well >>> >>> lets move closer to my location, us-east-2 Ohio (same instance type >> and os) >>> >>> grabbing ubuntu from that same mirror >>> >>> 2022-06-17 17:31:00 (29.5 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’ >> saved [3654957056/3654957056] >>> >>> all right, that has lost quite a bit of steam, but it's >>> still nothing to sneeze at, just to be sure lets grab >>> an iso off the Princeton university mirror 120MB/s ok.. >>> good enough... >>> >>> onto cloning blender >>> >>> Receiving objects: 5% (104235/2078173), 25.44 MiB | 692.00 KiB/s >>> >>> well... that's not looking promising... lets wait it out >>> >>> cloning from git.blender.org <http://git.blender.org> 17m23.449s >>> >>> sadface... lets try svn next, i lost patience and did not let >>> it finish.. iftop reported a top RX of 5.59Mbit though.. >>> >>> To summarize: >>> >>> Ubuntu iso from mirror.nl.datapacket.com < >> http://mirror.nl.datapacket.com> (Speed taken from wget) >>> My home - western Canada - 1.3 MB/s >>> AWS - London - 232 MB/S >>> AWS - Ohio - 29.5 MB/s >>> >>> Clone of git.blender.org <http://git.blender.org> (time taken from >> linux time command) >>> My home - western Canada - 27m39 >>> AWS - London - 2m32 >>> AWS - Ohio - 17m23 >>> >>> SVN Clone: (peak RX bitrate taken from iftop) >>> My home - western Canada - 1.5 Mbit/s >>> AWS - London - 64 Mbit/s >>> AWS - Ohio - 5.59 Mbit/s >>> >>> Think the only thing we really can conclude is >>> that being further from the server is leading >>> to an "unhappy" time for the developer. Given >>> the fact the EC2 instances were 100% identical >>> between London and Ohio, it's unlikely to be a >>> client configuration issue. >>> >>> --Ray >>> >>> On 2022-06-17 10:10 a.m., Dan McGrath via Bf-committers wrote: >>> > Hi, >>> > >>> > Really, I would need to see a capture from both sides to dig into >> it. I >>> > need to see what is being sent, and what is received on each side. >> There >>> > could be some weird firewall stuff happening that is causing >> fragmentation >>> > or blocking TCP options, or window scaling issues, etc. >>> > >>> > As for the default settings for tortoisesvn, the connection stuff >> I was >>> > referring to was about TCP/IP behaviour, rather than an application >>> > setting. Either way, I need to see what is actually happening. The >> fact >>> > that I can pull 5-8MB/sec (~40mbit) from Canada and not have a >> single >>> > interruption is telling. Granted, the host only has around 100 or >> 200mbit >>> > allocated, iirc. Multiply that by a few dozen people and you have >> a problem. >>> > >>> > For a simpler test, maybe try a large svn check, perhaps some old >> FreeBSD >>> > ports svn repo if you can find one, something also in .NL, and see >> how it >>> > goes, just to help eliminate your PC/firewall. Worst case >> scenario, you can >>> > also poke me in chat and we can try some server side and client >> side >>> > captures. >>> > >>> > TBH though, 1gig uplink just isn't a whole lot to serve our user >> base ;) >>> > >>> > On Fri, Jun 17, 2022 at 11:45 AM Tom M <[email protected]> >> wrote: >>> > >>> >> On Fri, Jun 17, 2022 at 5:20 AM Dan McGrath < >> [email protected]> >>> >> wrote: >>> >>> Hi Tom, >>> >>> >>> >>> Long time no see :) >>> >>> >>> >>> I did some tests from home and found that, aside from slow >> speeds, I was >>> >> able to checkout at least 5 or 6 gig of the bf-blender repo >> without error >>> >> or having to retransmit. >>> >> >>> >> Yeah, looks like I was mistaking really slow download of the >> larger >>> >> (.7 GB) libraries as freezing. I was averaging about .3 MB/s. I >> used >>> >> a different SVN client and saw that it was indeed downloading (the >>> >> sliksvn doesn't provide any feedback on files in progress, only >> once a >>> >> file completes) - It was taking about 40 minutes for each of the >>> >> larger files. It did complete eventually after about 5 hours. >>> >> >>> >>> I took the liberty of packet capturing the stream from my PC's >> POV, and >>> >> placed them in the ftp download area[1], if you care to take a >> look. >>> >>> The gist is that I was able to get around 5MB/sec for at least >> 5G of the >>> >> checkout, before I cancelled it. >>> >> Interesting, I usually get up to 3 MB/s download here normally >> from >>> >> other sources, but was typically getting .3 MB/s, with occasional >>> >> bursts of higher speed to a max of .8 MB/s when it wasn't >> downloading >>> >> the large files. >>> >> >>> >>> Now, my IP is a bit of a special case, so I am wondering if you >> are just >>> >> hitting the server so hard with http requests in separate >> sockets, or >>> >> something extreme, that is causing the system to block you for a >> period of >>> >> time. For comparison, my PC only had about 4 sockets open to the >> server for >>> >> the transfer. I would be curious to know if you are doing >> pipelining, or >>> >> sending each request in a separate connection. >>> >> Whatever the sliksvn and tortoisesvn defaults are, I'd assume >> they are >>> >> setup for typical usage. >>> >> >>> >>> If you can, the next time it happens, see if you are unable to >> visit >>> >> Phabricator or resume the svn checkout process. It may not be a >> firewall >>> >> issue, but without seeing some packet captures, I wouldn't be >> able to >>> >> really guess. Ideally, try to capture a few packets, until >> failure. At the >>> >> very least, you will need to snap 20b+20b=40b for IP+TCP headers, >> but I >>> >> would suggest a tad higher, maybe 64b. >>> >>> Cheers, >>> >>> >>> >>> Dan >>> >>> >>> >>> [1] - https://download.blender.org/ftp/dan/svn-checkout/ >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Jun 16, 2022 at 2:43 PM Tom M via Bf-committers < >>> >> [email protected]> wrote: >>> >>>> When checking out the libraries from SVN following the >> directions on >>> >>>> >>> >>>> https://wiki.blender.org/wiki/Building_Blender/Windows >>> >>>> >>> >>>> The svn checkout of the libraries fails with significant >> frequency, >>> >>>> >>> >>>> if you google you might stumble across using make svnfix >>> >>>> >>> >>>> >> https://www.mail-archive.com/[email protected]/msg151996.html >>> >>>> >>> >>>> Even after using it, I'm still having to do Ctrl C, and then >> make >>> >>>> svnfix repeatedly (it downloads some files, than randomly >> freezes, >>> >>>> sometimes after 20-50 files, sometimes after 100's of files). >>> >>>> >>> >>>> Git doesn't seem to have any problems, though. >>> >>>> >>> >>>> So might we consider migrating the libs to git, rather than >> using SVN? >>> >>>> >>> >>>> LetterRip >>> >>>> _______________________________________________ >>> >>>> Bf-committers mailing list >>> >>>> [email protected] >>> >>>> List details, subscription details or unsubscribe: >>> >>>> https://lists.blender.org/mailman/listinfo/bf-committers >>> >>> >>> >>> >>> >>> -- >>> >>> Cheers, >>> >>> Danny >>> >>> >>> >>> ---------------------------------------------------------- >>> >>> Danny McGrath - [email protected] >>> >>> GPG key: EDF6 AFF5 2086 F93A 1F59 36A5 44B6 26F3 6968 71CA >>> > >>> _______________________________________________ >>> Bf-committers mailing list >>> [email protected] >>> List details, subscription details or unsubscribe: >>> https://lists.blender.org/mailman/listinfo/bf-committers >>> >> _______________________________________________ >> Bf-committers mailing list >> [email protected] >> List details, subscription details or unsubscribe: >> https://lists.blender.org/mailman/listinfo/bf-committers >> > > _______________________________________________ Bf-committers mailing list [email protected] List details, subscription details or unsubscribe: https://lists.blender.org/mailman/listinfo/bf-committers