Re: REST impact paper

> On Sep 9, 2017, at 2:33 PM, Sean Conner <[email protected]> wrote:
> 
>  And I'm sure there are assumptions you made writing that code that open
> source, meant to be used who knows where, can't make.  One probably being
> that 95% of all JSON blobs being parsed are below a certain size so you can
> optimize for that.  Or that they only have a certain nesting level.  It also
> sounds like you did not write a drop-in replacement but had to rewrite the
> code to handle the new JSON parser.


Yes, wasn’t a drop-in replacement. It needed a different architecture. The document sizes themselves were highly variable even within single data sources (probably about 5 orders of magnitude range typically). It was ultimately used on a very diverse set of schemas over time but I can’t say much about it since I didn’t write it. 


>  Question:  how many of the tests presented here:
> 
> 	http://seriot.ch/parsing_json.html
> 
> would your high performance, proprietary JSON parser pass?


No idea but we did an enormous amount of conformance testing, including tests like the above. If it was broken then probably no more so than other popular JSON parsers. It never gave us problems with any data source we saw.

The only problem I recall was people deploying the SSE4 build on machines ancient enough that they didn’t have SSE4, which we hadn’t anticipated. Easy enough to fix though.  


>  For PostgreSQL, that would be an I/O scehduler, right?  Or perhaps direct
> disk I/O?  The engineering resources might be beyond the scope of what
> PostgreSQL can handle.  


PostgreSQL is an interesting case. The fact that a significant percentage of all major commercial enterprise databases are PostgreSQL forks (Vertica, Netezza, Greenplum, Redshift, Aster, etc) is a testament to what is possible in theory. They also have plenty of engineering resources working on it; they might lack expertise for some optimizations but not manpower. 

Modifications to PostgreSQL are usually required to be incremental to ensure the stability and reliability that it is famous for. However, it has several deep architectural limitations that make it difficult to modernize if done incrementally. For example, PostgreSQL can only support very small (by modern standards) page sizes that makes it impractical to implement many broad performance optimizations. Fixing this is not trivial, as increasing the representable page size causes a cascade of data structure breakage throughout the code base. This would be a big change but it is well-defined and something that could be rolled into a release.

However, increasing the supportable page size is not free. It would create a noticeable performance regression in isolation, which is also not acceptable. It enables large optimizations but is not by itself an optimization. Implementing the optimizations that can take advantage of it to offset the performance loss are themselves substantial changes to the PostgreSQL code base. At this point, the risk to users of unexpected and undesirable behavior due to the cascade of architectural changes is deemed too high. People who want to take those risks fork (and usually close) the code base and call the database something else.


Adding support for a proper I/O subsystem is a similar story, with an added aversion to the amount of arcane OS-specific code involved. I can understand this; this significantly increases testing complexity and the amount of specialized knowledge that current contributors may not have required to work on the code base. 


>  (Then there might be operating system limitations---Linus Tovalds, for
> instance, doesn't think highly of direct disk I/O, so Linux doesn't have
> great (or any) support for that.  Good luck fixing that)


Linux 2.6.x added great direct disk I/O support at the insistence of database developers. I’ve used it in most software I’ve built for a decade. 

Linus didn’t like the idea but the performance benefit for database engines is undeniable. As I recall, his objection was more that you could (and should) effect similar results by reinventing mmap() and friends with richer/better APIs and capabilities, though that would violate POSIX conformance. It would make it far easier for even relatively naive databases to implement efficient storage engines, which is a noble goal for Linux.


Andrew


_______________________________________________
FoRK mailing list
http://xent.com/mailman/listinfo/fork