How Spark and Hadoop Work

Hadoop began out of Yahoo as the technology that they developed to index the web. You take your data set and you split it across a bunch of machines. The reason these technologies have been successful is because if you can figure out how to fit what you're doing into MapReduce, it's actually a very easy way to program.