AWS Morning Brief

Links:CodeBuild to exfiltrate data from an AWS VPC: https://awsteele.com/blog/2022/02/03/aws-vpc-data-exfiltration-using-codebuild.htmlThousands of Open Databases: https://InfoSecwriteups.com/how-i-discovered-thousands-of-open-databases-on-aws-764729aa7f32“Why do Amazon S3 Data Breaches Keep Happening?”: https://markn.ca/2022/why-do-amazon-s3-data-breaches-keep-happening/You’re going to be placed on a public list of shame: https://Twitter.com/0xdabbad00/status/1489305680490106880?s=12How to report security issues in other people’s software: https://Twitter.com/notdurson/status/1489350457730469888S3 Bucket Negligence Award: https://www.zdnet.com/article/unsecured-aws-server-exposed-airport-employee-records-3tb-in-data/“Security Practices in AWS Multi-Tenant SaaS Environments”: https://aws.amazon.com/blogs/security/security-practices-in-aws-multi-tenant-saas-environments/Stratus Red Team: https://github.com/Datadog/stratus-red-teamTranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They’ve also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That’s S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Hello there. Another week, another erosion of the perception of AWS’s hard security boundaries. I don’t like what 2022 is doing to my opinion of AWS’s security track record. Let’s get into it.We start this week with a rather disturbing post from Aidan Steele, who talks about using CodeBuild to exfiltrate data from an AWS VPC. We’re increasingly seeing increased VPC complexity, which in turn means that most of us don’t have a full understanding of where the security boundaries and guarantees lie.Someone decided to scan a bunch of public AWS IP ranges and lo and behold, an awful lot of us suck at security. Specifically, they found Thousands of Open Databases. This is clearly not an exclusively AWS problem seeing as how it falls fairly on the customer side of the Shared Responsibility Model, but it does have the potential to be interpreted otherwise by folks with a less nuanced understanding.Mark Nunnikhoven has a blog post up that asks the question “Why do Amazon S3 Data Breaches Keep Happening?” I’ve often wondered the same thing. The vector has been known for years, the console screams at you if you attempt to configure things this way, and at this point, there’s really little excuse for a customer making these mistakes. And yet they keep happening.Scott Piper has had enough. He’s issued a simple warning: If you’re a vendor who offers a solution that deploys EC2 instances to customer environments, and you don’t support IMDSv2, you’re going to be placed on a public list of shame. He’s right: His first shame example is AWS themselves with a new feature release. For those who aren’t aware of what IMDSv2 is, it’s the instance metadata service. Ideally, you have to authenticate against that thing before just grabbing data off of it. This is partially how Capital One wound up getting smacked a couple years back.Corey: You know the drill: You’re just barely falling asleep and you’re jolted awake by an emergency page. That’s right, it’s your night on call, and this is the bad kind of Call of Duty. The good news is, is that you’ve got New Relic, so you can quickly run down the incident checklist and find the problem. You have an errors inbox that tells you that Lambdas are good, RUM is good, but something’s up in APM. So, you click the error and find the deployment marker where it all began. Dig deeper, there’s another set of errors. What is it? Of course, it’s Kubernetes, starting after an update. You ask that team to roll back and bam, problem solved. That’s the value of combining 16 different monitoring products into a single platform: You can pinpoint issues down to the line of code quickly. That’s why the Dev and Ops teams at DoorDash, GitHub, Epic Games, and more than 14,000 other companies use New Relic. The next late-night call is just waiting to happen, so get New Relic before it starts. And you can get access to the whole New Relic platform at 100 gigabytes of data free, forever, with no credit card. Visit newrelic.com/morningbrief that’s newrelic.com/morningbrief.Corey: AWS’s Dan Urson has a thread on how to report security issues in other people’s software. Something about it’s been nagging at me, and I think I’ve figured out what it is. Ignore the stuff about, “Have a coherent report,” and, “Demonstrate a reproduction case;” it gets into following the vendor’s procedures and whatnot around disclosure. I think it has to do with where I’m coming from. I generally don’t find security problems, or other bugs, by actively exploiting vendor systems; instead, I trip over them as a customer trying to get something done. The idea that I owe that vendor much of anything when I’m in that position rankles a bit. I get that this is a nuanced topic.And of course, 3TB of airport employee records were exposed in this week’s S3 Bucket Negligence Award. I hate to sound like I’m overly naive here, but what exactly is in the employee records that makes them take up that much space? I’m a big believer in not storing information you don’t need, and that just seems like an enormous pile of data to have lying around awaiting compromise.AWS themselves had an interesting post go out: “Security Practices in AWS Multi-Te...

Want to give your ears a break and read this as an article? You’re looking for this link.https://www.lastweekinaws.com/blog/guardduty-for-eks-and-why-security-should-be-freeNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of February 7, 2022 with Corey Quinn.

Links:Three vulnerabilities: https://blog.wiz.io/black-hat-2021-aws-cross-account-vulnerabilities-how-isolated-is-your-cloud-environment/Embarrassingly long time: https://Twitter.com/christophetd/status/1486610249045925890“Companies Leave Vast Amounts of Sensitive Data Unprotected”: https://www.propublica.org/article/identity-theft-surged-during-the-pandemic-heres-where-a-lot-of-the-stolen-data-came-from?token=pIt-Qx8lrKMcPei_lM3rFDQpHXkkcxXQGoogle Drive started mistakenly flagging files as infringing copyright: https://www.theregister.com/2022/01/25/google_drive_copyright_infringement/“How to deploy AWS Network Firewall to help protect your network from malware”: https://aws.amazon.com/blogs/security/how-to-deploy-aws-network-firewall-to-help-protect-your-network-from-malware/“How to use tokenization to improve data security and reduce audit scope”: https://aws.amazon.com/blogs/security/how-to-use-tokenization-to-improve-data-security-and-reduce-audit-scope/“Ransomware-resistant backups with S3”: https://www.franzoni.eu/ransomware-resistant-backups/TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They’ve also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That’s S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.After the content for this episode was effectively laid out, AWS did a late Friday night announcement of a new GuardDuty enhancement that would automatically opt people in to a chargeable service unless they explicitly opted each account out. This obviously doesn’t thrill me or other affected customers. so, as I record this, the situation is still evolving, but rest assured I’m going to have further thoughts on this next week.Now, let’s see what happened last week in AWS security. so, last year, Wiz found three vulnerabilities that allowed attackers to read or write into other customers’ AWS accounts. This flew beneath the radar at the time, but they’re all coming out of the woodwork now, and AWS’s security reputation, more or less, lies in tatters, replaced by a reputation for clamming up and admitting nothing. I’m already wincing at this summer’s re:Inforce keynote. if they try their usual messaging line, it’s not going to end well for them.There was apparently a serious vulnerability within the Linux polkit library. It took Amazon Linux an embarrassingly long time to acknowledge it and put out a release. Now, I’m not a fan of single-vendor Linux installs; any bets on how many non-Amazonians have commit rights to the distribution?Failing to learn from experience is never a great look, but as per ProPublica, “Companies Leave Vast Amounts of Sensitive Data Unprotected” despite decades of breaches. Please, please, please, if you’re listening to this, don’t be one of them. There’s no value in buying the latest whiz-bang vendor software to defend against state-level actors if you’re going to leave the S3 bucket containing the backups open to the world.And an uncomfortable reminder that we might not be the only parties perusing our “private” files stored within various cloud providers, Google Drive started mistakenly flagging files as infringing copyright. Now, amusingly the files in question tended to consist entirely of a single character within the file, but the reminder isn’t usually something that cloud providers want dangled in front of us. Once again we are, in fact, reminded that Google considers privacy to be keeping information between you and Google.Corey: You know the drill: you’re just barely falling asleep and you’re jolted awake by an emergency page. That’s right, it’s your night on call, and this is the bad kind of Call of Duty. The good news is, is that you’ve got New Relic, so you can quickly run down the incident checklist and find the problem. You have an errors inbox that tells you that Lambdas are good, RUM is good, but something’s up in APM. So, you click the error and find the deployment marker where it all began. Dig deeper, there’s another set of errors. What is it? Of course, it’s Kubernetes, starting after an update. You ask that team to roll back and bam, problem solved. That’s the value of combining 16 different monitoring products into a single platform: you can pinpoint issues down to the line of code quickly. That’s why the Dev and Ops teams at DoorDash, GitHub, Epic Games, and more than 14,000 other companies use New Relic. The next late-night call is just waiting to happen, so get New Relic before it starts. And you can get access to the whole New Relic platform at 100 gigabytes of data free, forever, with no credit card. Visit newrelic.com/morningbrief that’s newrelic.com/morningbrief.AWS had a couple interesting blog posts. One of them was “How to deploy AWS Network Firewall to help protect your network from malware”. and I’m torn on this service, to be honest, because On the one hand, it extends the already annoying pricing model of the Managed NAT Gateway, but On the other, it provides a lot more than simple address translation and is cost-competitive with a number of other solutions in this space. I think I’m going to land on, “use it if it makes sense for you, but don’t expect it to be cheap.”And a great blog post from AWS security folks—which is, honestly, something I have said a lot in the past, and I look forward to saying a lot more of in the future—

Want to give your ears a break and read this as an article? You’re looking for this link.https://www.lastweekinaws.com/blog/going-out-to-play-with-the-cdkNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 31, 2022 with Corey Quinn.

Links:GitHub organizations: https://alsmola.medium.com/securing-github-organizations-9c33c850638CloudTrail would spew other accounts’ credentials your way: https://onecloudplease.com/blog/security-september-cataclysms-in-the-cloud-formationsSpot on: https://research.nccgroup.com/2022/01/13/10-real-world-stories-of-how-weve-compromised-ci-cd-pipelines/Some excellent points: https://www.darkreading.com/cloud/enterprises-are-sailing-into-a-perfect-storm-of-cloud-risk“Amazon EC2 customers can now use ED25519 keys for authentication with EC2 Instance Connect”: https://aws.amazon.com/about-aws/whats-new/2022/01/ed25519-keys-authentication-ec2-instance-connect/“Integrating AWS Security Hub, IBM Netcool, and ServiceNow, to Secure Large Client Deployments”: https://aws.amazon.com/blogs/apn/integrating-aws-security-hub-ibm-netcool-and-servicenow-to-secure-large-client-deployments/“Best practices for cross-Region aggregation of security findings”: https://aws.amazon.com/blogs/security/best-practices-for-cross-region-aggregation-of-security-findings/Assume AWS IAM Roles using SAML.to in GitHub Actions: https://github.com/saml-to/assume-aws-role-actionTranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They’ve also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That’s S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: So, most interesting this week is probably my request for AWS to support a different breed of SSH key. No, it’s not a joke. Listen on and we’ll get there.So, from the security community last week, everyone talks about how to secure AWS environments. This post takes a different direction and talks about how to secure GitHub organizations, which makes sense if you think about it as an area to focus on. If you compromise an org’s GitHub repositories, it’s basically game over for that company.I also came across this post from 2020, talking about how if asked politely, CloudTrail would spew other accounts’ credentials your way. How many more exploits like this have we seen and just never been told about?NCC Group has some great stories up about compromising CI/CD pipelines, and they are all spot on. Because nobody really thinks about the Jenkins box that has everyone working with it, outsized permissions, and of course, no oversight.Enterprise cloud risk is a very real thing, so a post from Josh Stella, who’s the CEO of Fwage—though he pronounces it as ‘Fugue’—and it makes some excellent points, and also cites me, so of course, I’m going to mention it here. We incentivize the behaviors we want to see more of. There’s a security lesson in there somewhere.Corey: This episode is sponsored in part by our friends atNew Relic. If you’re like most environments, you probably have an incredibly complicated architecture, which means that monitoring it is going to take a dozen different tools. And then we get into the advanced stuff. We all have been there and know that pain, or will learn it shortly, and New Relic wants to change that. They’ve designed everything you need in one platform with pricing that’s simple and straightforward, and that means no more counting hosts. You also can get one user and a hundred gigabytes a month, totally free. To learn more, visitnewrelic.com. Observability made simple.Now, from AWS, what have they said? “Amazon EC2 customers can now use ED25519 keys for authentication with EC2 Instance Connect”. I really wish they’d add support for ECDSA keys as well, and no, this is not me making a joke. Those are the only key types Apple lets you store in the Secure Enclave on Macs that support it, and as a result, you can use that while never exporting the private key. I try very hard to avoid having private key material resident on disk, and that would make it one step easier.“Integrating AWS Security Hub, IBM Netcool, and ServiceNow, to Secure Large Client Deployments”. I keep talking about how if it’s not simple, it’s very hard to secure. AWS, IBM, and ServiceNow, all integrating is about as far from “Simple” as is possible to get.“Best practices for cross-Region aggregation of security findings”. And this was a post that I was about to snark that it should be as simple as “Click the button,” but then I read my post, and to my surprise and yes, delight, it already is. Good work.And in the land of tool, I found a post talking about how to assume AWS IAM Roles using SAML.to in GitHub Actions, and I really wish that that was first-party, but I’ll take what I can get. Because again, I despise the idea of permanent IAM credentials just hanging out in GitHub or on disk or, realistically, anywhere. I like these ephemeral approaches. You can be a lot more dynamic with it and breaching those credentials doesn’t generally result in disaster for everyone. And that’s what happened last week in AWS security.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or whereve...

Want to give your ears a break and read this as an article? You’re looking for this link.https://www.lastweekinaws.com/blog/clickopsNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 24, 2022 with Corey Quinn.

Links:S3 Bucket Negligence Award: http://saharareporters.com/2022/01/10/exclusive-hacker-breaks-nimc-server-steals-over-three-million-national-identity-numbersAnyone in a VPC, any VPC, anywhere: https://Twitter.com/santosh_ankr/status/1481387630973493251A disgruntled developer corrupts their own NPM libs ‘colors’ and ‘faker’, breaking thousands of apps: https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/“Top ten security best practices for securing backups in AWS”: https://aws.amazon.com/blogs/security/top-10-security-best-practices-for-securing-backups-in-aws/Glue: https://aws.amazon.com/security/security-bulletins/AWS-2022-002/CloudFormation: https://aws.amazon.com/security/security-bulletins/AWS-2022-001/S3-credentials: https://simonwillison.net/2022/Jan/18/weeknotes/TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by my friends at Thinkst Canary. Most companies find out way too late that they’ve been breached. Thinkst Canary changes this and I love how they do it. Deploy canaries and canary tokens in minutes, and then forget about them. What’s great is then attackers tip their hand by touching them, giving you one alert, when it matters. I use it myself and I only remember this when I get the weekly update with a, “We’re still here, so you’re aware,” from them. It’s glorious. There is zero admin overhead to this, there are effectively no false positives unless I do something foolish. Canaries are deployed and loved on all seven continents. You can check out what people are saying atcanary.love. And, their Kube config canary token is new and completely free as well. You can do an awful lot without paying them a dime, which is one of the things I love about them. It is useful stuff and not a, “Oh, I wish I had money.” It is spectacular. Take a look. That'scanary.love because it’s genuinely rare to find a security product that people talk about in terms of love. It really is a neat thing to see.Canary.love. Thank you to Thinkst Canary for their support of my ridiculous, ridiculous nonsense.Corey: So, yesterday’s episode put the boots to AWS, not so much for the issues that Orca Security uncovered, but rather for its poor communication around the topic. Now that that’s done, let’s look at the more mundane news from last week’s cloud world. Every day is a new page around here, full of opportunity and possibility in equal measure.This week’s S3 Bucket Negligence Award goes to the Nigerian government for exposing millions of their citizens to a third party who most assuredly did not follow coordinated disclosure guidelines. Whoops.There’s an interesting tweet, and exploring it is still unfolding at time of this writing, but it looks that making an API Gateway ‘Private’ doesn’t mean, “To your VPCs,” but rather, “To anyone in a VPC, any VPC, anywhere.” This is evocative of the way that, “Any Authenticated AWS User,” for S3 buckets caused massive permissions issues industry-wide.And a periodic and growing concern is one of software supply chain—which is a fancy way of saying, “We’re all built on giant dependency chains”—what happens when, say, a disgruntled developer corrupts their own NPM libs ‘colors’ and ‘faker’, breaking thousands of apps across the industry, including some of the AWS SDKs? How do we manage that risk? How do we keep developers gruntled?Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers.Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That’s goteleport.com.AWS had a couple of interesting things. The first is “Top ten security best practices for securing backups in AWS”. People really don’t consider the security implications of their backups anywhere near seriously enough. It’s not ‘live’ but it’s still got—by definition—a full set of your data just waiting to be harvested by nefarious types. Be careful with that.And of course, AWS had two security bulletins, one about its Glue issues, one about its CloudFormation issues. The former allowed cross-account access to other tenants. In theory. In practice, AWS did the responsible thing and kept every access event logged, going back for the full five years of the service’s life. That’s remarkably impressive.And lastly, I found an interesting tool called S3-credentials last week, and what it does is it helps generate tightly-scoped IAM policies that were previously limited to a single S3 bucket, but now are limited to a single prefix within that bucket. You can also make those credential sets incredibly short-lived. More things like this, please. I just tend to over-scope things way too much. And that’s what happened Last Week in AWS: Security. Please feel free to reach out and tell me exactly what my problem is.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—a...