Networking in the Cloud Fundamentals: BGP Revisited with Ivan Pepelnjak

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Transcript
Corey: Hello and welcome to our Networking In The Cloud mini series, sponsored by ThousandEyes. That's right. There may be just one of you, but there are a thousand eyes. On a more serious note, ThousandEyes has sponsored their cloud performance benchmarking report for 2019, at the end of last year, talking about what it looks like when you race various cloud providers. They looked at all the big cloud providers and determined what does performance look like from an end user perspective? What does the user experience look like among and between different cloud providers? To get your copy of this report, you can visit snark.cloud/realclouds. Why real clouds? Well, because they raced AWS, Azure, GCP, IBM Cloud, and Alibaba, all of which are real clouds. They did not include Oracle cloud because, once again, they are real clouds. Check out your copy of the report at snark.cloud/realclouds.

Welcome to week 12 of the Networking In The Cloud mini series of the AWS Morning Brief, sponsored by ThousandEyes. So one of the early episodes of the Networking In The Cloud mini series had me opining and relatively uninformed broad brush strokes about the nature of BGP. Today I am joined by Ivan Pepelnjak, who is a former CCIE who wrote a fascinating blog post that I will link to in the show notes, saying, "This is great, but this is what happens when someone who's good at one thing steps completely out of their comfort zone into things they don't fully understand and start opining confidently, if not authoritatively." Ivan, thank you for taking the time to speak with me.

Ivan: Thanks for having me on. And no, I was way more polite than your summary.

Corey: Absolutely. I believe that there's a way to tell a story of the hero's journey that everyone talks about when they're building a narrative arc. Instead, I go for the moron's journey and I always like to be the moron because, generally, I tend to be, and as I walk through the world and get things sometimes right, occasionally wrong, I love being corrected when I stumble blindly into an area I don't know. First because it gives me an opportunity to learn something new, which is great, but it also gives me that opportunity to be the dumbest person in the room again, which is awesome. So...

Ivan: That's exactly why I blog to get your opinions.

Corey: Exactly. You have data, I have opinions and mine are louder seems to be the way that discourse works in the modern era. So from a high level, what did I get wrong about BGP?

Ivan: Well, you got everything right about the mess that we are in and the fragility of the generic internet infrastructure. The only thing you got wrong was that you blamed the tool, but not people using the tool.

Corey: It always feels like it's safer, on some level, to blame technology because if the takeaway is, "Well, the user experience around tool X isn't great, and that adds a contributing factor to why things break." That seems to be a message that carries slightly better than, "And thus the answer is for everyone to be smarter and stop screwing up." And that may very well be the answer. It's just a bitter pill to swallow sometimes. So I find blaming a tool is easy.

Ivan: Yeah, but it's like blaming the knives for people to get cut or blaming the chainsaw for people to cut off their arm because they were not properly trained.

Corey: One of my assertions was that BGP is more or less a hot mess because it was designed for an era when people on the internet fundamentally could trust one another and that doesn't seem to be the case today. The analogy in my mind, that I don't think I mentioned, was SMTP, the the email protocol, for lack of a better term. When that was built, the internet was more or less comprised of researchers and who in the world would ever abuse a protocol like email? It's not like there was any money involved in the internet. Fast forward today and your spam folder is inherently a garbage fire.

Ivan: Yeah, but BGP has a slightly different history. It was redesigned a few times. There were several attempts to get the global routing protocol right. And BGP, the last attempt, already included the tools that allow entities that don't trust each other, like commercial internet service providers, to exchange information and apply policies on inbound and outbound updates. So for example, I don't want to hear about your customers because I hate you and I don't want to peer with you or I don't want to tell you about my customer because that customer has a special deal and their traffic can only go through some other transit providers so I will not tell you about that customer. Those things were already a major requirement when BGP was designed and it always included the tools to implement the policies that individual commercial entities wanted to have, which by the way, never happens to SMTP. We have BGP version 4 now and we are still on SMTP version zero.one plus enhancements.

Corey: I guess the best analogy I can come up with through my exposure with BGP, because I tend to handle inter networking between various groups about as well as I write code, things that I have some vague awareness that there are things you should be doing here that I will almost certainly not get right, so I back away slowly and leave it to professionals. As a result, every time I really see how BGP works in any hands-on sense or a point where it's forced upon my awareness, it's similar to how I become aware of plumbing. I don't think about it. I don't question it. I just expect when I turn the faucet on or flush the toilet that water will do what it's going to do. I don't expect the toilet to explode. So the only time I think about BGP is when there is a peering dispute or when there's a flap or, on one notable occasion, when I was at a security conference and, as a demo, some folks hijacked the entire AS of the SN for the conference and rerouted it halfway around the world and back, which explained why everything was super latent and crappy.

Ivan: Yeah. You're absolutely right, but all the incidents you mentioned are not the fault of the tool. They are the fault of the tool not being properly used. And also, let's be honest, it took them hundreds of years to get the plan being to the point when you can just turn on the faucet and the clean and drinkable water comes out of it. It's not like that would have happened in the last year or two, and very probably it wouldn't have happened without public pressure to bring us dri...