Luciano and Eoin explore the wonderful world of data streaming using Kafka on AWS. In this episode we focus mainly on Managed Streaming for Kafka (or MSK) and discuss what are the main differences between MSK and Kinesis. We also explore the main features that MSK provides, its scaling characteristics, pricing and, finally, how MSK works in conjunction with other AWS services.
We conclude the episode by providing a decision tree that should help you to decide whether you should use Kinesis or MSK or avoid streaming services entirely in favor of something like SNS or SQS.
In this episode, we mentioned the following resources:
- Our previous episode on Kinesis data streams: https://www.youtube.com/watch?v=u_nR6up4Kvs
- Our series of Event services: https://www.youtube.com/watch?v=CG7uhkKftoY&list=PLAWXFhe0N1vLHkGO1ZIWW_SZpturHBiE_
- AWS MSK sizing spreadsheet: https://dy7oqpxkwhskb.cloudfront.net/MSK_Sizing_Pricing.xlsx
- Should My Startup use Kinesis or MSK? - https://www.youtube.com/watch?v=TJS19EuzH2k
- Intro to MSK (reinvent talk from 2018) - https://www.youtube.com/watch?v=9nKswHsLseY
- Running Apache Kafka on AWS (by Frank Munz) - https://www.youtube.com/watch?v=HtU9pb18g5Q
- Cloudonaut - Kinesis versus MSK - https://www.youtube.com/watch?v=kcBAKz0MPf8
This episode is also available on YouTube: https://www.youtube.com/AWSBites
You can listen to AWS Bites wherever you get your podcasts:
- Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017
- Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q
- Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw==
- Breaker: https://www.breaker.audio/aws-bites
- RSS: https://anchor.fm/s/6a3312a0/podcast/rss
Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on Twitter:
- https://twitter.com/eoins
- https://twitter.com/loige