[CAnet - news] The power behind Google

For more information on this item please visit the CANARIE CA*net 4 Optical
Internet program web site at http://www.canarie.ca/canet4/library/list.html
-------------------------------------------


[Some excerpts from HPCwire article-
http://www.taborcommunications.com/hpcwire.html--BSA]

  
  "Few Web services require as much computation per request as search 
  engines. On average, a single query on Google reads hundreds of 
  megabytes of data and consumes tens of billions of CPU cycles. 
  Supporting a peak request stream of thousands of queries per second 
  requires an infrastructure comparable in size to that of the largest 
  supercomputer installations." So begins the description of the 
  computational power required for Google's Web search engine in "Web 
  Search For A Planet: The Google Cluster Architecture", a publication 
  of the IEEE Computer Society.
  
  
  
  HPCwire: Could you briefly describe the Google computing 
  infrastructure and its rationale?
  
  Dean: When designing our computing clusters, we place a great deal of 
  emphasis on what sort of systems will give us the best 
  price/performance. Search applications are relatively easy to 
  parallelize, both within processing of a single query (by partitioning 
  the index across machines), and across queries (by replicating each 
  piece of the index across multiple machines and having each replica 
  serve a fraction of the total traffic). Given this easy parallelism, 
  the price/performance argument leads towards using clusters of large 
  numbers of commodity PCs, that is, x86 processors, inexpensive hard 
  drives, commodity Ethernet networking, etc.
  
  Our clusters typically are composed of several thousands of these 
  commodity machines, all connected via commodity Ethernet. The 
  individual machines typically have gigabit NICs, and groups of around 
  40 machines are connected to commodity gigabit Ethernet switches. 
  These switches are then connected into a large-scale core switch for 
  the cluster, using a small number of Gigabit connections per group of 
  40 machines. 
  
  Having lots of relatively small machines means that you get a lot more 
  bang for the computing dollar, but it also means that the machines are 
  less reliable than more expensive machines, and because there are so 
  many of them, higher-level software has to be designed to tolerate 
  failures -- with thousands of machines, machine failures happen many 
  times per day. Our software is designed to assume that the hardware 
  can fail.  Once you do that, it becomes fairly simple to deal with a 
  lot of failures. Our serving systems generally have multiple replicas 
  for each piece of the system to provide fault tolerance to individual 
  machine failures.
  
  We've also designed our own file system, the Google File System (GFS), 
  to reliably store large amounts of data on large clusters of machines. 
  [For information on GFS visit 
  <http://labs.google.com/papers/gfs.html>.]
  
  Finally, when you're doing large-scale data processing, it's important 
  not to separate the storage from where you're going to do the 
  computation. You don't need really high-end storage arrays with 
  massive amounts of bandwidth to process a large amount of data. If you 
  do the scheduling right, you can read the data from local disks on 
  thousands of machines, simultaneously. By doing this, you can attain 
  really good bandwidth from low-end storage systems with slightly 
  clever software.  So rather than moving the data to the machine, we 
  try to move the computation to the data.
  

  My colleague, Luiz Barroso, has written up a nice article describing 
  why CMP processors look very attractive for our applications [see 
  <http://labs.google.com/papers/priceofperformance.html>].
  
  As always, we're continually evaluating and refining our hardware 
  infrastructure to explore which solutions provide the most attractive 
  price/performance for our applications, but we are excited about the 
  initial CMP processors coming out, as we feel their emphasis on high 
  throughput for parallel applications rather than single-thread 
  performance is a good match for our applications.
  
  HPCwire: Could you describe the MapReduce model and implementation and 
  how it is being used within Google?
  
  Dean: MapReduce is a system originally developed by myself and my 
  colleague, Sanjay Ghemawat, as a way of describing computations that 
  want to process input data to compute some derived data. The general 
  programming model is to break the computation down into two distinct 
  phases: a Map phase, and a Reduce phase. Users specify a Map function 
  that processes a key/value pair to generate a set of intermediate 
  key/value pairs, and a Reduce function that merges all intermediate 
  values associated with the same intermediate key. The basic idea is 
  similar to the Map and Reduce primitives found in LISP and many other 
  functional languages.
  
  What makes it interesting is that we've developed a MapReduce library 
  that is able to take programs written in this style and make them run 
  on clusters of hundreds or thousands of machines
  
  We've been pleasantly surprised at how applicable the general 
  MapReduce model has been to a wide variety of problems: it's being 
  used internally at Google in areas as diverse as our core crawling and 
  indexing system, data mining, statistical machine translation, our 
  advertising systems, processing of satellite imagery, etc. It makes it 
  relatively easy for people within Google to write relatively simple 
  code and have that code run reasonably efficiently on thousands of 
  machines. In a typical day at Google, thousands of different MapReduce 
  computations are run with hundreds of distinct Map and Reduce 
  functions across our various computational clusters.
  
  HPCwire: Is MapReduce something you would make publicly available?
  
  Dean: We don't currently make it available. We've had thoughts about 
  it. It would be a moderate amount of effort on our part to divorce it 
  from other pieces of our software, such as our cluster scheduling 
  system. Also, we put a fair amount of effort into it and it's not 
  clear that we would want to make it available to our competitors. At 
  the same time, we feel like there are a lot of academic projects that 
  would benefit from having access to something like this. So, in the 
  future, I wouldn't be surprised if we did make it available in some 
  form. [For more information about MapReduce visit 
  <http://labs.google.com/papers/mapreduce.html>.]
  
  
  
  Many of our newer products, like Gmail and Google Earth, have fairly 
  different characteristics than Web searching, and it's important to 
  have infrastructure and system building blocks that meet the needs of 
  a diverse set of products that we want to offer, and to make it easy 
  to develop new applications and services. One example is that we saw a 
  need in many of our products to manage large amounts of 
  semi-structured mutable data in interesting ways, and to help with 
  that, we're developing BigTable, a large-scale distributed storage 
  system for managing semi-structured data.
  
  In the future, a combination of both new products and our goals for 
  pushing our current products in new and interesting directions will 
  guide our decisions about the right software tools and infrastructure 
  to build. These are very exciting times to be working on large-scale 
  systems and products at Google.
  

-------------------------------------
To SUBSCRIBE:
send a blank e-mail message to
[email protected]

To UNSUBSCRIBE:
send a blank email message to
[email protected]
-------------------------------------

These news items and comments are mine alone and do not necessarily reflect
those  of the CANARIE board or management.



-----------
[email protected]
www.canarie.ca/~bstarn
skype: pocketpro
SkypeIn: +1 614 441-9603


_______________________________________________
news mailing list
[email protected]
http://lists.canarie.ca/mailman/listinfo/news