Re: REST impact paper
"Stephen D. Williams" <[email protected]>
| Newsgroups | gmane.culture.people.rohit-khare |
|---|---|
| Message-ID | <[email protected]> |
On 9/9/17 12:13 PM, J. Andrew Rogers wrote: >> On Sep 8, 2017, at 10:10 PM, Sean Conner <[email protected]> wrote: >> >> .... >> So what magic things were done to make JSON parsing faster with >> proprietary code than with open source code? Or alternatively, what does >> open source do wrong? I'm asking for specific examples and not a >> generalized "it sucks" argument. > > It is just a difference in priorities. Operational cost is not even in the top three priorities for open source projects when building software, so when there is a conflict between priorities it loses out. These journeys always start out with using the high-performance open source library du jour but end up with proprietary implementations. > > For JSON specifically, I needed to parse trillions of documents as quickly as possible, so performance was rarely satisfactory in any case. Because “open source”, we could inspect the architecture and implementation for ways to improve performance. Being software performance junkies, we find many opportunities to materially improve throughput, some algorithmic and some architectural. > > Open source projects will often only accept patches for a subset of algorithmic optimizations, rejecting some because they violate some other aesthetic or goal of the project (e.g. it is an anti-optimization for some obscure or archaic hardware). Architectural optimizations are effectively a non-starter for an existing open source project even when it would be hugely beneficial (e.g. PostgreSQL). The projects aren’t wrong, they just value many things above performance, and it is their project. > > We ended up implementing our own highly-optimized JSON parser written in C++11. It was much faster than the open source ones it replaced. Nothing fancy, just tight engineering that prioritized performance. We went back later and reimplemented certain hot paths in assembly using SSE4 for systems that supported it, for another 2.5x improvement in throughput as I recall. Github repo or it didn't happen. (I have my own queue of things to publish...) > > The lack of attention to operational cost in open source is no less of a problem than the lack of attention to user interface design. Users care about these things a lot even if programmers tend not to. > > Andrew sdw _______________________________________________ FoRK mailing list http://xent.com/mailman/listinfo/fork