The Swyx Mixtape cover image

[Tech] What is Hadoop? - Jared Hillam

The Swyx Mixtape

00:00

Hadoop vs SQL - Should I Hold Up the Whole Response?

As each copy of my Twitter data finishes its assigned segment of history the answers are delivered to a reducer on a separate server cluster. This cluster is responsible for adding up the tallies or producing a consolidated list. If one of those 60 plus servers breaks down or has some sort of issue while processing my request, Hadoop would say no and it would provide the user with an immediate answer. The eventual consistency methodology in Hadoop is a far more realistic method of reading continuously updating feeds of unstructured data across thousands of servers. For example Facebook created something called Hive which allowed their team members that didn't know how to write Java code to write queries in standard SQL.

Transcript
Play full episode

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
App store bannerPlay store banner
Get the app