AWS Morning Brief Whiteboard Confessional: Help, I’ve Lost My MFA Device!
About Corey Quinn
Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.
Links
Transcript
Corey: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.
This episode is sponsored by a personal favorite: Retool. Retool allows you to build fully functional tools for your business in hours, not days or weeks. No front end frameworks to figure out or access controls to manage; just ship the tools that will move your business forward fast. Okay, let's talk about what this really is. It's Visual Basic for interfaces. Say I needed a tool to, I don't know, assemble a whole bunch of links into a weekly sarcastic newsletter that I send to everyone. I can drag various components onto a canvas: buttons, checkboxes, tables, etc. Then I can wire all of those things up to queries with all kinds of different parameters, post, get, put, delete, etc. It all connects to virtually every database natively, or you can do what I did and build a whole crap ton of lambda functions, shove them behind some API’s gateway and use that instead. It speaks MySQL, Postgres, Dynamo—not Route 53 in a notable oversight; but nothing's perfect. Any given component then lets me tell it which query to run when I invoke it. Then it lets me wire up all of those disparate APIs into sensible interfaces. And I don't know frontend; that's the most important part here: Retool is transformational for those of us who aren't front end types. It unlocks a capability I didn't have until I found this product. I honestly haven't been this enthusiastic about a tool for a long time. Sure they're sponsoring this, but I'm also a customer and a super happy one at that. Learn more and try it for free at retool.com/lastweekinaws. That's retool.com/lastweekinaws, and tell them Corey sent you because they are about to be hearing way more from me.
Welcome to the AWS Morning Brief: Whiteboard Confessional. Today I want to talk about infosec. Specifically, an aspect of infosec that I think is not given proper attention, namely two-factor auth. Now, two-factor auth is important to enable but first, back up a second. Use a password manager with strong passwords for all of your stuff. Those are table stakes at this point.
Now, most password managers will offer to also store your multi-factor auth codes, your OTP tokens, etcetera. I'm not a big fan of that because it feels to me, perhaps incorrectly, like I'm collapsing multiple factors back down into that same factor. Someone gets access to my password manager, worst-case scenario, I’m potentially hosed. That's not great. Now, the password managers will argue that this isn't technically true, yada, yada. I'm old fashioned. I'm grumpy. I'm an old Unix systems administrator that had certain angry loud opinions, so I'm going to keep using separate tools for managing passwords, as well as getting in as a second factor. May I also point out that SMS is terrible as far as a second factor. Don't use it if you possibly can, for reasons that go well beyond the scope of this show: we're not that kind of podcast.
Now, let's talk about what happens if you, for one reason or another, lose your MFA device, or the app on your phone because this happened to a certain business partner of mine named Mike Julian. Now, Mike wound up getting a new phone, which is great because his was something from the Stone Age presumably some kind of Nokia candy bar phone. I hear someone dropped one of those things once the last time they were in mass sale and accidentally killed the dinosaurs. So, that's the kind of era of phone he was upgrading from to, I think, the iPhone SE, but don't quote me on that. I don't tend to pay attention to his taste in electronics. Personally, I question his taste in business partners, but that's all right; he signed on the dotted line; he stuck with me now.
So, he inadvertently wound up losing access to all of his old MFA tokens and having to get them re-added in other places. Some systems worked super well for this. It was a matter of, “Oh, I'll just use my backup codes,” which he kept like a good responsible person. It let him in, he would then be able to regenerate backup codes, change over the device and everything was glory. For others, he wasn't so lucky and had to phone in and get a reset after identity verification. So, now he didn't have his multi-factor device, so it would fall back to using SMS because it had his cell phone. And he could not disable that with some environments. So, that becomes an attack vector, if you're able to compromise an SMS number which, surprise, is not that hard if you put some effort into it.
This, of course, does bring us to Amazon. Mike needed to reset his Amazon MFA token. Now, when I say Amazon, I don't mean AWS. I mean, Amazon, and I'm going to go back and forth as I go down the story a little bit. So, this is an Amazon retail account, not an AWS account. And it turns out when you Google how to reset your Amazon MFA token, all the results are about AWS.
So, “Okay, that's interesting,” says Mike. He Googles effectively to remove all results from aws.amazon.com. Cool. Now all the results are about things that are not Amazon stuff. Not anything helpful. So, there's no documentation in Google for any of this as applies to Amazon retail, it may as well not exist as a problem. This is less than ideal from Mike's perspective. He was able to reset his AWS multi-factor auth for the AWS account—that's for the same email address tied to that amazon.com account, but AWS and Amazon have completely separate MFA infrastructures.
So, this is fascinating. He posts on Twitter, which is the number one way to get help when you have an Amazon issue and you run a company devoted to making fun of Amazon, and AWS support chimes in because they're helpful. Someone else says, “I've been trying to solve this problem for 10 years and got nowhere. Good luck, Godspeed.” And it seemed odd because it's an Amazon retail problem. Why is AWS chiming in? And this leads to a phone call. Mike finally winds up getting on a series of phone calls with AWS support.
...