
[Tech] What is Hadoop? - Jared Hillam
The Swyx Mixtape
00:00
Hadoop vs SQL - Should I Hold Up the Whole Response?
As each copy of my Twitter data finishes its assigned segment of history the answers are delivered to a reducer on a separate server cluster. This cluster is responsible for adding up the tallies or producing a consolidated list. If one of those 60 plus servers breaks down or has some sort of issue while processing my request, Hadoop would say no and it would provide the user with an immediate answer. The eventual consistency methodology in Hadoop is a far more realistic method of reading continuously updating feeds of unstructured data across thousands of servers. For example Facebook created something called Hive which allowed their team members that didn't know how to write Java code to write queries in standard SQL.
Transcript
Play full episode