Defensive Security Podcast Episode 277
Defensive Security Podcast - Malware, Hacking, Cyber Security & Infosec
Navigating Kernel Security: Trade-offs and Tensions
This chapter explores the critical role of kernel-level operations in cybersecurity, focusing on malware detection and prevention methods used by firms like CrowdStrike and Sentinel-1. It contrasts the security approaches of Microsoft and Apple, emphasizing the complexities in their ecosystems while also touching on a significant cybersecurity event and a Microsoft summit aimed at enhancing endpoint security. The discussion extends into real-world implications, including a case involving North Korean operatives that underscores the urgent national security challenges posed by cyber threats.
In this episode, Jerry Bell and Andrew Kalat discuss various topics in the cybersecurity landscape, including the influence of cyber insurance on risk reduction for companies and how insurers offer guidance to lower risks. They touch upon the potential challenges with cybersecurity maturity in organizations and the consultant effect. The episode also goes into detail about issues surrounding kernel-level access of security tools, implications of a CrowdStrike outage, and upcoming changes by Microsoft to address these issues. They recount a case about a North Korean operation involving a laptop farm to gain employment in U.S. companies, posing major security concerns. The discussion highlights the pitfalls of relying on end-of-life software, especially in M&A scenarios, and how this could be a significant vulnerability. Lastly, they explore the massive data breaches from Snowflake and the shared security responsibilities between service providers and customers, emphasizing the importance of multi-factor authentication and proper security management.
Links:
https://www.cybersecuritydive.com/news/insurance-cyber-risk-reduction/724852/
https://www.cybersecuritydive.com/news/snowflake-security-responsibility-customers/724994/
Transcript:
Jerry: Here we go. Today is Saturday, August 24th, and this is episode 277 of the defensive security podcast. My name is Jerry Bell and joining me today as always is Mr. Andrew Kalat.
Andrew: Good evening, my good sir Jerry. How are you?
Jerry: I am awesome. How are you?
Andrew: I’m good. I’m good. I’m getting ready for a little bit of a vacation coming up next week So a little bit of senioritis. If I’m starting to check out on the show, you’ll know why
Jerry: Congrats and earned. I know.
Andrew: Thank you, but otherwise doing great and happy to be here as always
Jerry: Good. Good deal. All right. Just a reminder that the thoughts and opinions we express on this show are ours and do not represent anyone else or including employers, cats, relatives, you name it.
Andrew: various sentient plants
Jerry: Exactly. Okay. So jumping into some stories today. First one comes from cybersecuritydive. com, which by the way, has a lot of surprisingly good content.
Andrew: Yeah, I have enjoyed a lot of what they write. We’ve a couple good stories there
Jerry: Yeah. Yeah. So the title here is insurance coverage drives cyber risk reduction for companies, researchers say that the gist of this story is that there were two recent studies done or reports released one from a company called Omeda and another one from Forrester, which I think we all know and love.
And I’ll summarize it and say that they’re both reports indicate that companies which have cyber insurance tend to be better at quote, reducing risk more likely detect, respond, and recover from data breaches and malicious attacks compared to organizations without coverage. So I thought that was a little interesting.
On the other hand it to me feels like a bit of availability bias, so by that, what I mean is if you go and take a survey of people who go to the gym and work out at the gym on their diet, you will probably will find out that Eat a healthier diet than the public at large.
Andrew: But I go.
Jerry: you just go.
Andrew: I, look,
Jerry: I’m not saying, I’m not saying everybody, right?
Andrew: least I show up, right? And I’ve been told showing up is half the battle.
Jerry: It is half the battle, that’s right. Knowing is the other half.
Then doing is the other half.
Andrew: I will say, speaking of G. I. Joe quotes, I thought catching on fire was going to be a far bigger problem in my life than it turned out to be.
Jerry: That and quicksand.
Andrew: I, we were
Lot about that as children of
Jerry: quick, quicksand.
Andrew: Heh.
Jerry: QuickSand was, I, I lived in fear of QuickSand, but it turns out it’s really not that big of a concern.
Andrew: For as much as I heard stop drop and roll done it
Jerry: Yet.
Andrew: That’s true. The day is young. Anyway back to your story. I think you’re right I will also say having worked with a number of these companies do interestingly have their own towards trying to keep you from getting hacks. They have to pay out So they do push certain things like and I’ve seen myself and I won’t say it You know, it doesn’t matter where, when, but if you have things like one of the well known EDR tools well deployed, they might cut you a rate on or a break on your rates. Because they have their actuarial table saying, Hey, if you’re using certain bits of technology that lowers your risk of usually ransomware, right? So they
Jerry: Sure.
Andrew: seems to me, my opinion is that these insurance companies feel that some of the well known EDR brands in a Windows environment It is very effective or decently effective at stopping ransomware, therefore they’re less likely to pay out, therefore they lower your rates. So there might be some of that too. They do to give companies guidance on what they see across their industry to reduce risk.
Jerry: I think that, that makes sense. I’ll say, on, on one hand, like I was saying before, I think companies that buy cyber insurance are probably maybe more mature, more invested in, protecting their environment than others. But I think that there’s also this consultant effect when when you want to drive change and whether whatever kind of change that is, reorganizing revamping your security program, justifying additional expenses for anything outside guidance, typically Carries a lot more weight than something that comes from internal.
Andrew: Sad but
Jerry: and so I think, yeah, anybody who’s been in the industry for a long time or really any amount of time knows that, especially this is a, the CISO trick, right? When you come into a new organization as a CISO, the first thing you do is you go off and you hire a, a big name consultant.
You burn a half a million bucks on a consulting engagement. And at that point, it’s not you telling the company, Hey, we’ve got to spend a bunch of money to improve our security program. It’s some, hard to argue with independent third party who is making that assessment. And to some extent you argue with that at your own peril, right?
Because now it’s it’s a, it’s an assessment that becomes exhibit a, if something goes wrong and which is, both a blessing and a curse. But my experience is it certainly helps a lot. And I think that this cyber insurance and their somewhat prescriptive guidance and expectations around the kinds of controls and technologies you need to have in place is a very similar kind of thing, right?
If you’re engaging with them, they’re going to be opinionated on what you should and shouldn’t be doing and and then like a consulting engagement. It’s a third party giving you that guidance. And so I think that tends to carry a lot more weight.
Andrew: Agreed on all points. The only caveat I would say to that is sometimes these recommendations that come from some insurance companies are not customized typically to your particular risk environment or situation. They are very broad approaches to reducing risk across many different types of environments with many different types of risk profiles. Technology stacks and all that sort of stuff. So they’re very somewhat generic recommendations, I think.
Jerry: I think you’re probably right. In any event, it’s I thought it was I thought it was quite interesting. Certainly having that insurance can help. I will tell you in my time as a CISO in dealing with customers and to some extent business partners, there was a I would say a growing expectation that you have to have cyber insurance.
Actually, I experienced firsthand quite a few customers actually writing into contracts. That you have now, I don’t know how far and wide that permeates the industry, but I think it’s probably becoming a lot more common these days because, companies have this interdependence and so it’s not necessarily just like a cloud service provider where that kind of thing can manifest, look at over the, what now, 12, 13 years we’ve been doing the show.
How many times have we talked about a company like, let’s say, Target or Home Depot getting hacked as a result of something happening with one of their suppliers? And so I think, as time goes on, we’re going to see that becoming kind of table stakes to, to have these business relationships, especially with larger and more mature companies.
Andrew: Why do you think that is, what do you think that the third party is assuming that you will get from that insurance? Just so you have the ability to recover from an incident and sustain As a going concern or that they assume that if you have insurance, it’s coming with requirements that level up the maturity of your program or what value do you think that third party sees in their business partner having cyber insurance?
Jerry: That’s a great question. I think it’s both, actually. I think there is this, naive view that if, if something bad were to happen this insurance would, provide that buffer. It would make sure that, the company didn’t go out of business, but the reality is that, especially, if you look at some of the really large hacks.
can happen with relatively small organizations who are, I would say fairly highly leveraged, at least in terms of their insurance policy. So yeah, it’s great. They may have a 5 million insurance policy, but if they hit if they’re, let’s say a, a hundred million dollar company and they get hit with a, 50 million in breach fees, their 5 million in insurance coverage, isn’t really going to go very far.
So I don’t know that it’s extraordinarily useful in terms of protecting customers from harm. I think there’s a facade that it provides. And I also think it does give some, at least a segment of, roles at companies gives them this warm, fuzzy feeling that somebody else is looking over their shoulders.
In that respect, it’s not different than like a sock to or an ISO, SOA or what have you.
Andrew: I wonder if there’s some sort of implied, Hey, you’ve ransomware you can recover faster. The other thing I think about is the perverse incentive. So when we look at an insurance in general, it’s to shift risk. It’s to shift
Jerry: I
Andrew: risk to a third party. So is there the risk that a executive committee will say, Hey, we don’t need to invest in much in cybersecurity because we have insurance if something bad happens.
Jerry: mean, I would love to sit here and say no, that’s that would never happen. But I don’t think it happens that every organization, but I definitely expect it happens more than it should.
Andrew: Yeah, it’s interesting. It’s interesting interplay of competing priorities. When you start to introduce these sorts of things and how what sort of behavioral economics comes into play
Jerry: Yeah, absolutely. All right. Anyway, go go talk to your insurance carrier and it might might help you with your internal program and justify additional improvements to your program. Our next story comes from Ars Technica and the title here is Crowdstrike.
Unhappy with shady commentary from competitors after outage
Andrew: I’m shocked. I say shocked
Jerry: Totally surprised by this so we’ve talked about this Several times and i’m sure we’ll talk about it several more times CrowdStrike obviously had a pretty devastating Snafu With one of its products that caused probably the largest single meltdown of I.
T. in history and a lot of their competitors have been capitalizing on that outage. And so now this story is talking about in the wake of some of the back and forth tit for tat. mudslinging that’s been going on. I think they call out Sentinel one in particular. CrowdStrike is, I think, getting a little peeved at how their competitors are behaving, basically saying, hey, this could have happened to anybody.
And I think there’s a lot of differing opinions in the industry based on my experience and exposure to different, to, the industry. I don’t think everybody’s on that bus. I think there’s a lot of people who think that, no, this really would be a lot less likely with other companies. Although it is interesting that SentinelOne is, is one, I think one of the more aggressive mudslingers, but they also, by the way, as far as I can tell, do use they do access windows The kernel.
And in fact, the next story we have actually talks directly about that.
Andrew: Yep they do and this goes back to something that I’ve I don’t have expertise in so I’m just dancing around and pontificating at something I can’t be authoritative on but I think what I keep seeing is that most security tooling feel that they need to be in the Windows kernel to be effective on the way Windows is architected today. it’s interesting when they talk about they being various competitors of CrowdStrike talk about safer methodologies, whatever that means, and I think somewhat that implies perhaps not operating at the kernel level. However, safer in terms of not causing an outage per se, but are they as effective at spotting and stopping malware? I don’t know. I, my assumption is there’s always some sort of trade off. If we’ve got most of the industry wanting to operate at the kernel level, and we’ve got another story that talks about this a little bit, and Microsoft themselves is talking about maybe we can find ways to make this effective. seems to me as not a, not having worked at those companies that, but Operating at the kernel level allows these security tools to be more resilient against malware trying to shut them down, and in theory be faster and more effective, and if they are operating at the user level or in user space, the implication that I’m getting from these articles is malware could Shut down the anti malware tool and do whatever it wants to do. And that appears to be harder at the kernel level. That it’s better able to protect itself and spot things at a deeper level in the operating system. I don’t know if that’s true, but it seems to be most of these companies operate that way. And in fact, there was even an implication we talked about it on a previous show. From Alex Stamos, who’s the newly appointed CSO over at or c SSO or cso, one of the two over at Sentel
Jerry: CTO. CTO.
Andrew: Now it says Chief Information Security Officer in this particular
Jerry: Oh, okay.
Andrew: Alright. Anyway, he talked about, Hey, we’ll back outta using the kernel if all of our competitors will as well. there’s clearly some advantage to being there. And I don’t know that anybody really wants to talk about that.
Jerry: I think there’s, I think there, that we are talking about this as if it’s like one, monolithic choice of, you’re either there or you’re not there.
Andrew: Yeah.
Jerry: that’s probably not the right way to think about it. I suspect that there’s the nuances that there are certain kinds of functions that you don’t need to perform in the kernel.
And as an example, you don’t need to, parse your file, your definition files in the kernel. You could do that in user mode and then. pull it into the kernel module. I suspect that there’s some of that. It certainly adds a lot of complexity and I’m not going to argue that, but I think that’s where, I don’t know if, I don’t know anything about SentinelOne and their technology, but I’m going to guess what, when they say, when they’re trying to throw rocks at CrowdStrike, I think what they’re probably saying is we do.
more things outside of outside of the kernel. But like you said, if they were to completely move out of the kernel their ability to function would be impaired. And so that kind of dovetails into the second story about this, which is from CNBC. And the title there is Microsoft plans September cybersecurity event to discuss changes after a CrowdStrike outage.
And there was I don’t know if this was directly an outgrowth of the comment that Ed Bastian, the CEO of Delta, made to CNBC, gosh, right after the outage. He said something like, we don’t see this sort of problem with Apple. And, and he was. really talking about, and by the way I’m assuming that Ed had heard that from, his security people hey, Apple just doesn’t allow this, which is true, right?
They don’t they are much stricter about this access. Microsoft will certainly point to the the decision that was rendered against them in the EU that forced them to open this up because by the way, Microsoft is a direct competitor to both SentinelOne and CrowdStrike when it comes to endpoint security products.
Like they have their own, they have their own product, which is in contrast to Apple. Apple doesn’t really have, an equivalent thing like Microsoft Defender, right? They,
Andrew: a single brand around security. They do things a little differently. This is a tough comparison, and I struggle I hear this, and I get frustrated because it’s In a vacuum that isn’t looking at market share. It isn’t looking at
Context. It isn’t looking at all the tradeoffs Microsoft had to make to run an open hardware ecosystem and all the backward compatibility choices they made Apple, to just say Apple just does it better.
It’s not and I’m not an anti Apple fan. I use Mac all day long, but it is Microsoft Mechanics www. microsoft. com. I disingenuous to say it outside of the context of everything else around that ecosystem that has contributed to this. And all the frustrations people have with Apple being so hardcore about, Oh yeah, you, sorry, this hardware no longer supported, go away. Or where is Apple’s surfer ecosystem or, there, there’s just, he’s not wrong, but it’s also like one 10th of the story.
Jerry: Sure. You
Andrew: like that. there’s, it’s not just, it’s not just Microsoft is stupid. It’s, and Apple is smart. There’s so much more that goes into this. It’s my frustration.
Jerry: know, as a society, though, we’ve boiled everything down to 15 second soundbites, it’s just the way of the world.
Andrew: You’re not
Jerry: lost our, we’ve lost our tolerance for nuance.
Andrew: But that’s
Jerry: But,
Andrew: Jerry. To bring the nuance back.
Jerry: That’s exactly right. Microsoft on September 10th, which is coming up fast. is going to have this summit with endpoint security providers. And I think what they’re trying to establish is a set of best practices around what is and is not done in the kernel so that they can avoid catastrophes like this going forward.
And then also, as we’ve talked about in the past, they’re going to start trying to encourage these companies to use the eBPF interface, which is, an alternative way of hooking into the kernel that has less ability. It provides probably not the exact same level of control and visibility, a very, very substantially similar without all some of the downside.
So I think I think Microsoft’s eBPF implementation is rapidly maturing as, relative to what’s been out there for Linux for some time, but it is, it’s not something that it might my experience that the security industry is really embraced yet, but I think this is probably going to be the forcing function that really drives us in that direction.
And this, by the way, may be the thing that It ultimately enables Microsoft to say, no more access to the kernel. If you want to do this, you’ve got to do it through this particular feature like eBPF. I think that’s how I see this playing out. I think if you were to look five years, five years down the road, I don’t think companies are going to be linking and hooking into the kernel.
I think they’re going to be forced through a function like eBPF. It’s just Jerry’s wild ass speculation though.
Andrew: It makes sense. One quote that I do want to mention from this article that kind of goes to what we were saying in the previous story, which is, quote, Software from CrowdStrike, Checkpoint, SentinelOne, and others in the endpoint protection market currently depend on kernel mode. Such access
Jerry: Right.
Andrew: quote, monitor and stop bad behavior and prevent malware from turning off security software, end quote, a spokesperson said. So that kind of
Jerry: Yeah.
Andrew: what we were saying earlier that there’s clearly some
Jerry: Yes.
Andrew: sure that they’re trying to get, like you mentioned, new methodology to give them that same capability without being deep in the kernel.
Jerry: But I think the, I think both customers of these endpoint products and the manufacturers of them too are going to say, okay, fine. But then tell me how you’re going to help us not get, not our endpoint security product, not get killed by. The ransomware companies the adversaries who are pretty adept at stopping things, even that are running in the kernel today, it’s a lot easier to do with user mode processes.
So I, I think that there’s going to be a bit of a meet me in the middle here where Microsoft is going to say, get your crap out of the kernel. And those companies are going to say then, give us something. Enable us because they’re, I don’t think they’re fully enabled today. Yes,
Andrew: wouldn’t be fair. And I think that’s where the EU is coming out with their ruling years ago that started all this. So there’s, make no mistake, I guess is what I’m saying here, that there is absolutely competitive standoff going on here. These companies are frenemies in this situation, so they don’t want to back out and be, leave Windows, their competitor with the only capability to do something that they can’t do, and then they can’t compete.
Jerry: Exactly. And by the way, my Microsoft we have, we talked about it before. We have seen instances where Microsoft has shot themselves in the foot too, right? This is, it’s not unprecedented that Microsoft’s own updates have caused outages. Outages. So I’ve got to, I’ve got to believe like they’re thinking about this too.
It’s so great. They they kick all the other endpoint security products out of the kernel. And then suddenly they are, they’re the cause of the next CrowdStrike scale outage because of their security product is now the only one that’s in the kernel. How bad would that be for them? Holy crap.
Anyway I think that’s what we’re going to see. Agreed.
Andrew: on you there.
Jerry: No, all good. So our next our next story is also from Ars Technica, and this is a bit of a follow up from the Know Before story that we talked about, I think, two or three episodes ago. The title is, title here is, Nashville Man Arrested for Running Laptop Farm to Get Jobs for North Koreans.
So the, if you recall,
If you recall the security awareness company Know Before published a blog post about how they had hired what turned out to be a North Korean agent. And the way that went down was they interviewed and selected and hired a a person who turned out to be a North Korean citizen. And they shipped the laptop to a laptop farm.
And I actually had some questions about how that worked. This actually explains how this goes down. It’s very enlightening. So this this person named Newt, his last name is Matthew Isaac Newt, who lived in in Nashville. So he had a relationship with a set of people in North Korea, where he would facilitate and it looks like it was a fairly sophisticated operation where he would help track down identities that they could use.
He provided a place for North Koreans who were being hired by, unknowingly hired by the way, by U. S. companies to have their laptop sent to. He would receive the laptops, put it on his home network. install remote access software and allow the North Koreans remote access into the laptop so they could quote, do their work.
And it’s just a, it’s a fascinating thing. He I guess they, they said that they were making each one of the North Korean employees was making about 250, 000 over the roughly year period that this was going on. The allegation by the US government is that the money these people were earning was in turn being used by North Korean to fund their weapons program.
So obviously not not super awesome. And they do go on to say in the article, by the way, this is not a one off situation. They refer to another person named Christina Marie Chapman. Down in Arizona who basically did the very same thing So it’s a fascinating I didn’t realize this was as large of a problem as it is, but Apparently this is a fairly Industry becoming an industrial scale operation run by, individuals.
I’m I’m surprised.
Andrew: Yeah it’s interesting. It’s a little different to the know before because in this case, it looks like the North Korean it employees are doing legitimate work. They’re not immediately installing, malicious software and that they’re trying to earn their wage as it were
Activity in this case.
But I’m also very curious how the money was moved over to North Korea. And, was this, paid, I’m sure it’s paid to a US bank account. And then what does it look like to get it over North Korea, which is somewhat non trivial. I don’t know if they use Bitcoin or something like that. But that’s a big part of the charges here is the wire fraud, that sort of thing.
But the other thing I think about is, if as an employer don’t allow your random employee to install software, it would stop a lot of this. I get that’s a big cultural taboo and there’s a lot of gnashing of teeth around that topic, but if they couldn’t install remote access tools or, you as an IT department or security department don’t monitor for those remote access tools, it certainly would stop a lot of this.
You’d be, It just wouldn’t work, unless some other methodology is found. A way to fight this it’s one more reason to not necessarily let local users have full admin rights.
Jerry: Even if you do, I think it’s very prudent to actively look for and block and investigate people who are installing remote access tools because like remote access tools. Whether it’s RDP or TeamViewer or, any of the myriad other software, that stuff has been the source of so many security incidents over the years.
And in fact, it’s one of the common ways that frauds, like just garden variety fraud is perpetrated where, the Windows help desk scam, where they’ll call you up and ultimately install some sort of remote access software to get into your system. So I think this is really important.
Now I will say, I think this particular set of scams was reliant on them installing this remote access software. But I think, a sophisticated, if they were if they were sophisticated, network KVMs are pretty cheap these days. So it’s not necessarily a home run to say we’re like, we’re fully protected because we don’t allow that we would certainly detect it.
But it is not, there are other ways around that. It’s another step. And I can imagine it would make, perhaps the would increase the cost and complexity of hosting these but probably not prohibitively. And, it goes back to you’ve got to, especially in a remote working situation, you’ve got to have good diligence on and good awareness of who you’re hiring.
And by the way, that I say that is somebody like full throated supportive remote work.
Andrew: Yeah, certainly, but I also feel like you’ve spent a lot of time thinking about this. Is this your new post retirement career? Are you setting up laptop
Jerry: But my, my kids have moved. My oldest son has moved out and someday soon my youngest will move out. So I’m trying to figure out, do I go the Airbnb or do I go the, hosting laptop farms? I don’t know yet.
Andrew: That explains the eight new air conditioning units you just added onto your house.
Jerry: Yeah. Then the Bitcoin mining, like you gotta diversify.
Andrew: You having free time is dangerous. I don’t know that this is a good thing at all.
Jerry: What is the saying about idle hands?
Andrew: Indeed.
Jerry: All right. The next next article comes from Dark Reading, and the title here is Why End of Life for Applications is the Beginning of Life for Hackers. This is a big problem. So the gist of this story is that end of life applications is a boon for threat actors. It’s they make reference to 35, 000 applications moving to end of life status over the course of the next year.
I think that’s probably optimistic. I think it depends on how you define application for sure, but I think the look, in my, my, my personal experience. This is probably one of the larger problems we have in IT. It just, we talked last time about patching and how nobody wants to, nobody wants to patch stuff.
But there are so many issues that come along with using end of life software. Not the least of which, by the way, is that Most vulnerability management programs are built around, subscribing to vendor alerts to understand that a patch was, needs to be applied. When you’re using end of life software, that doesn’t happen anymore.
Like it’s, it goes quiet. Certainly.
Andrew: end of life for vulnerabilities. Much less issue patches.
Jerry: Yeah, exactly.
Andrew: Yeah.
Jerry: Yeah, most of the time you’re. Your vendors, vendor notifications are not vulnerability based, they’re patch based. They announce the availability of a patch to fix a vulnerability. And so now you know that you have a piece of work that has to be done because a patch was released and you got to go and apply that patch.
There’s no patch. A lot of vulnerability scanners work in a similar way, especially as it pertains to maybe less less well known applications. Now, certainly if you’re using a tenable Nessus and you’re running an out of support version of Linux or Windows, like it, it flags that itself as a critical vulnerability, but you don’t have any granularity about what the actual technical vulnerabilities are because they don’t know.
It’s just like it’s end of life. Who knows what. What sort of vulnerabilities there are and I think it gets It starts to descend into obscurity after that when you get into like open source Components and whatnot. You just don’t know you’re unaware that they’re not being maintained anymore And that becomes a big problem I will also say, one of the things that they talk a little bit about how companies can get into the Or defend against letting this, letting things go end of life.
But I will say my, in my experience, it’s easy. It’s a trap to fall into when something goes end of life, right? Because, Hey it’s end of life, but there’s no known vulnerabilities and we have this other thing that needs to be done and it’s super high priority. It’s going to make a billion dollars, blah, blah, blah.
And at the time, that’s true, right? It’s true. It’s a low, it’s a low risk thing. Your your version of WordPress is out of date. There’s no known vulnerabilities. But then, you start to collect these things. And suddenly, you’re buried under too much technical debt. And and it’s hard to, really hard to get out from under and you end up in this position where you’ve got so much of this debt that you have a hard time even understanding, not only what is, what all is end of life, but, are there actually vulnerabilities?
Like at the time you made that risk acceptance to allow this end of life thing happen. You knew that there wasn’t, but are you actually keeping up with the vendor and with the industry to know whether that has changed? And I think, if you’re talking about one thing, one application, it’s fairly easy to manage.
But once you start accumulating a lot of these, it becomes really unmanageable.
Andrew: Yeah, you echoed a lot of the notes I had as well, which is once you get so far behind, it’s that much more difficult to get caught back up. And then it becomes that much more of a fight comparing against other higher priority things to work on patching or massive upgrading, as opposed to just keeping things up to date bit by bit. other thing I think about, I don’t have my notes here, is If something is that far end of life, it’s not just a security thing. You don’t know necessarily if other interactive or interrelated components are supporting that version anymore, or tested against that version. And you might start seeing some weird buggy artifacts as a result. and you brought up the open source thing. That’s. Most of these sort of end of life checks are usually coming from some sort of end of life policy statement from a commercially supported application or operating system. The problem with a lot of these open source dependencies and third party packages, they don’t have that.
They don’t have any sort of published end of life, end of support methodology. So how do you know when something goes end of life when it’s open source? It hasn’t been updated in three years. Is that just because it’s just really stable or because it’s been abandoned and what is the criteria you want to use there?
Do you have a criteria that says, Hey, we need to use currently maintained third party dependencies in our code. Okay. How do you define what that is? Something that has had an update in the last X amount of months we see all the time that. Open source packages just become abandoned ware without any notification, without any understanding of that.
It’s just in hindsight you go, oh yeah, that guy stopped working on that three years ago. knows.
Jerry: Yeah, it’s not certainly not universally true. There are plenty of open source, especially the larger ones that have a defined roadmap. But I think you’re spot on. If you don’t, if you don’t see that there’s been an update on a particular piece of open source, is it because there’s nothing wrong with it or, or is it on you to go look at its GitHub repository and see that like people have been.
jamming the issue log with requests to fix some vulnerability that are falling on deaf ears. And, when you start to think about the many tens of thousands or even hundreds of thousands of open source components, some applications use that, it becomes really challenging. And I think that’s where some of the open source management tools Like mend and others can start to help but you know Even those aren’t infallible and even so you have to have as an organization Some amount of discipline and capacity to make sure that you’re staying up to speed
Andrew: Yeah.
Jerry: The other thing by the way that I wanted to hammer on because I hate it.
I hate it With the burning passion and it has happened every time i’ve been involved in it And it’s something that I feel like as an industry, we have to do something about. And that is in the aftermath of an acquisition, accounting systems. Can we just talk about those for a second? Because I don’t know how many acquisitions I’ve been, on the acquired side and I’ve been on the acquire side a bunch of times.
And it happens every single goddamn time. The acquired company has an accounting system. And what happens when, why do companies acquire other companies? One of the main reasons is this thing called synergy, right? And the synergy basically means that we don’t want to run duplicate HR systems and accounting and whatnot.
If we consolidate all that backend stuff. And continue to make money and the products they sell, then like we have, we’ve increased the value that the whole is more than the sum of the parts. And that’s why a lot of companies
Andrew: Is,
Jerry: other,
Andrew: yeah. Sorry,
Jerry: right?
Andrew: gonna say the back office overhead
Jerry: And
Andrew: Yeah.
Yeah.
Jerry: exactly. So want to. When a company acquires another company, one of the things they want to do very quickly, as quickly as possible, is start to realize those savings. And one of the first things that happens is, putting out to pasture the accounting system. And this could also be the HR system, although, nobody does HR internally anymore.
It’s like running your own voicemail system these days. But accounting systems seem to be still very much insourced. And, every single time I, without exception, every single time I’ve seen this happen, company gets acquired. We stopped, they stopped paying for maintenance on the accounting application.
And. Ends up running on some old ass operating system that is out of date. It can’t be that you can’t update the operating system because the accounting system won’t work on the new operating system. You can’t update the app, the accounting system because like they’re out of business or you don’t have a license to upgrade anymore and it would cost millions of dollars and it’s not in the business case, my, oh my God.
And it. wouldn’t provide any like accretive value to the company if you did upgrade it because it’s not going to be used anymore it but you know what if you shut it off like people will go to jail at least that’s what i’ve been told over and over again and so you end up with this thing sitting in the corner which is as best i can describe it an attractive nuisance because like everything about it is terrible it’s all out of date and it can’t go away because you Joe from accounting says that somebody will go to jail if you turn it off.
Anyway, I may be a little angry about this. Ah,
Andrew: If you just ran your accounting on Excel, like God intended, you wouldn’t have this problem.
Jerry: it’s
Andrew: No
Jerry: so true.
Andrew: than what can be done in an Excel sheet.
Jerry: Amen. I, the only thing I can say is in, in those instances, and I look I’m going to just be forthright and say, I’ve never seen an effective. counter to this. It’s been a problem every single time I’ve seen it happen. The only thing I can say is, as an idea, have a, have a way I don’t think any company is going to be I shouldn’t say any company, but I think most companies will not be effective in changing those facts.
It’s going to happen, your acquired company is going to have an accounting system. There’s not going to be an appetite to update it but there it is, and so you have to you have to mitigate the risk of it, and I think that having a defined approach to doing that, whether it’s like a separate VLAN that has no access, or like you have to, do multi factor authentication to get in and out of that network, it could be pretty simple and dirty, but have a plan, because it just, It happens no matter how mad it makes me, it, it happens.
And and so I think we’ve got to recognize that there are cases where that will be and come up with, relatively workable mitigations around it, but it can’t be the rule.
Andrew: What’s your ideal use case? That they just migrate all the data off the old system to the new system and just kill it?
Because I’m
Jerry: I don’t know. I
Andrew: system is maintained, is because they need a system of record for the last seven years or whatever, for tax purposes or government regulatory purposes. And I’m guessing that’s why, I’m assuming.
Jerry: typically, yes. In, in, frankly, I think the best course would be to figure out the different types of reports that are needed and to do, to run exports and have those exports exist in a spreadsheet. Now, I don’t know if there, I’m not an accountant, I’m not a,
Andrew: right out of the See, look, it all comes back.
Jerry: But I don’t know if there’s some like statutory requirement that data, that system of record has to be there because if, like the sec or, the department of justice or some other legal Authority came to you and wanted to investigate like why did you claim? Why did you say you made x dollars in, seven quarters ago before you were acquired?
You’ve got to be able to go back and replay that maybe that’s why and maybe that can’t be done through exports I don’t know but I In every instance, the accounting folk have insisted that system has to be available. It’s not enough to just dump the data.
Now I don’t know if that’s laziness or what. The other problem I have, and while I’m beating on this drum, Over time, the people who are familiar with that system go away
Andrew: Certainly.
Jerry: and so like at one of the, one of the things you have to watch out for is that eventually what, like when that system’s usefulness is done, there isn’t anybody left to say, Hey, now it’s time to turn it off.
Andrew: And then who wants to take the risk of being the one who makes the wrong call? So they say people go to jail. Can we talk about who might go to jail and if that’s really a bad thing?
Jerry: I like where this is going. I,
Andrew: just weighing the outcome. What are my options?
Jerry: I think I’ve beat that one to death. The the last story we have today, it’s also from the cyber security dive. And the title is after a wave of attacks, Snowflake insists security burden rests with customers. Now, Snowflake had a a large problem. And this happened earlier in, in 2024 before we got back to podcasting.
But I would say that I don’t think it’s an overstatement to say that the security breaches or data breaches associated with Snowflake will probably go down as the largest in history. It, bigger than anything there ever was, and perhaps bigger than anything there ever will be again, maybe it’s a huge numbers of customers.
Lost lots of data. Now, the point of this story is to say, Hey, like Snowflake was not, snowflake is saying, Hey, it wasn’t us. It wasn’t us. It was. Our customers.
Andrew: Because they
Jerry: they’re not
Andrew: single factor login, was easy to find in other dumps of passwords bad actors did a widespread campaign to, and upward force, password test, all of those passwords against Snowflake user accounts and lo and behold, a bunch of them worked. And that’s, they’re
Jerry: Yes.
Andrew: shouldn’t have relied on single factor and reusing of passwords.
And that’s on you because you didn’t take appropriate security measures.
Jerry: Correct. Yeah we just provided you with an account and a place to store your data or process your data. It was on you, the customer, to pick a good password. And that’s basically what they are saying here. They’re saying our systems, our Snowflake systems didn’t get hacked. We, you know Our infrastructure is fine.
Everything worked as designed. The fact that bad guys got your password and stole all your data is a horrible thing, but it’s not our fault. It’s your fault because, it was your password that they got. We don’t know how they got your password. Was it the same password you used on LinkedIn or on, Ashley Madison?
I don’t know. Who, who knows? And that’s, they’re saying it’s not our problem, not our fault. And they basically, of course they do give lip service to the fact that, hey, there are customers and of course we care about them and we’re in it together as they say, but not our fault.
By the way they have since implemented some snowflake, I should say, has since implemented some changes, which require mandatory multi factor authentication for new customers, and it also gives customers the ability to require or enforce multi factor authentication for all of their users or for specific roles in their account.
So I should, by the way, I should have said for those of you not in the know, Snowflake is a what I would call like a managed storage managed database provider. They do lots of. Value added services around data analytics and whatnot. So the kinds of data that you would have stored in Snowflake are the kinds of data that you wouldn’t want to get compromised.
And so I think this was a central place, one stop shopping for some adversaries to go and do their password stuffing. And it looks like they they got somewhere in the neighborhood of 150 or 200 different customer accounts and pulled all that data out. And it was, and still is by the way, we’re still, even now hearing about net new companies that were hacked or had their data stolen.
And I don’t know if that’s because they’ve, been quote, responding and investigating to the incident, or if they just recently realized that this has happened, but this is a big problem and I think more interesting than, this itself, because I don’t think that Snowflake is all that, commonly used in the industry is the concept that, these service providers are.
They have a hard line of demarcation of what they’re responsible for. And so when we as consumers of these services decide what we’re going to go use, what do we do? Like we look at their SOC 2 and we look at their PCI report and we look at their ISO certification and we look, we look for all these things, but how often do we look to see What the capabilities of their services from a security perspective do you know, do you require multi factor authentication as by default, how many customers, I can say this authoritatively, I don’t know, in my time that I ever saw any customers asking about that, and they should.
And this is, the, because the, look go ahead.
Andrew: no I think this absolutely is a product management decision. And I
Jerry: Yes.
Andrew: the implication here is security causes friction. Friction
Jerry: Yes.
Andrew: make us uncompetitive. So if I, let me bring it back to a bank as an example. A bank knows how to secure your accounts. A bank could easily force multifactor authentication.
They could force tokens. They could force all sorts of things. But they know there’s a certain percentage of customers who will move away from them as a result. That friction will be more complicated than they want to deal with. And they will not value the security They will look at this as a detriment to that service and go someplace else. So that is a decision that every company and every product manager makes about what security they built into their tool. So in my mind obviously I have no insight, insider insight to Snowflake, but the idea that those admins could allow single user passwords and static passwords is a product design choice to reduce friction and usability of to make the usability of the platform easier. Whether it’s for whatever reason, as opposed to like multi factor authentication is a very mature known. Solved problem. So if it’s not being put in place, it’s because somebody made a choice and that choice typically is competitive in nature. So to your point,
Are not demanding it, it’s not going to show up. The other thing that this occurs to while I’ve stolen the floor for a moment is with these rise of all of these SaaS vendors, typically the administrators for these are no longer it professionals or security professionals. They are. users of the data professionals. So they may not even understand or have any insight into the implications of the administration of these tools around security aspects like this. All they know is, Hey, I need to make an account. Okay. I made an account. They may not have the background or the guidance to know single factors bad, and here’s why. Like you would, if a. More traditional IT or security team were administering these tools. So I think we’ve also got a problem with these SaaS tools that have become so ubiquitous and easy to use that we’ve somewhat enabled less technical staff to administer them and I think things like this become oversights that come back to bite us.
Jerry: I couldn’t have said it better. I think that’s exactly what’s going on. And so it’s not friction on IT and security departments that a product might require multi factor authentication. It’s friction on the business users. And so when the bit when it’s the business users that are specifying and deciding what services to use.
You know that. It’s not hard to imagine that they’ll pick one that is easier to log into even though it could have a devastating effect, like a lot of the customers of Snowflake here. And so it’s a complicated thing because if all of this, if all of the providers were requiring multi factor authentication, that wouldn’t be a, there wouldn’t be a difference.
It wouldn’t be a differentiation. I guess the business users would have kind of a similar experience across all of the different providers, but we know that’s, we know that’s not true. But I think that if you zoom out, the concern I have as an industry is that we’re we’re not broadly speaking.
We’re not. deeply aware of the responsibilities that we are picking up for properly managing those services. We are, to some extent, doing what we think is due diligence, looking at those providers and saying they’re a reputable company, they’re secure, they have these certifications, but then that’s like the end of it.
We don’t think about What our obligations are. And this manifests itself in so many different ways. Like how many times we talked about open S3 buckets. It’s another permutation of that. We had the big Capital One breach in AWS. Also misconfiguring how they set up their IAM.
Like it. The devil is in the details in how we manage these software as a service systems and, that the companies are not that these providers aren’t going to come to us and hit us on the back of the head and say, you big dummy, like you should have, you, we saw that you don’t have multi factor authentication turned on and you really need to go do that.
Now, maybe. snowflake will start doing that because of the reputation damage that they’ve incurred as a result of this breach, which they assert isn’t their fault. It is having an impact on them. It is, I think it is probably having a negative impact on them.
It’s attracting attention that I’m sure they don’t want to have. And I’ll, another way of thinking about it is like, there’s no such thing as bad PR, but I think You don’t want to have your name associated with the largest data breaches that have been around. But again,
Andrew: podcast like this with tens of listeners
Jerry: tens of listeners. And I’m not blaming, I’m not disagreeing by the way, I’m not disagreeing with the premise of Snowflake’s comments that it was, their customers were responsible. This was not intended to be like a Bash on Snowflake segment. It was more like we have to understand how companies like Snowflake are viewing their relationship with their customers.
You as the customer are responsible for ensuring that you are properly securing your stuff.
Andrew: So I go back to maybe we’ve got non technical running this, these tools in companies, maybe what we should be doing is allocating GRC’s time to go auditing how they’re running these tools
Jerry: Yes.
Andrew: and clean them up.
Jerry: I I don’t have a better, I don’t have a better option or better idea. Yeah,
Andrew: out there for and such that sort of are meant to be, allow you to apply policy, and those are technical controls of the same problem, which is proper policy should be applied. And I think the challenge is we have such a sprawl of SAS tools that it’s really difficult to stand on top of and ahead of. And I’ve worked at try to mandate as best they can. Hey, if you want to buy a SaaS tool, it’s got to integrate to single sign on or MFA. you could do that and then 10 minutes later, the admin sets up another username and password that you don’t have purview into. And they don’t necessarily know they’re doing anything wrong.
Jerry: exactly. They have a business objective to meet. They’re trying to solve a business problem.
So
Andrew: this is I think one of the unintended consequences of the, Democratization of admin capabilities through the SaaS and Cloud Sprawl, that is making life more difficult for security teams.
Jerry: Yes. And I do wonder, by the way I don’t know that we’ll ever have clarity on this, but I do wonder of the, I think it’s 165 at last count of the 165 companies who’s data was breached from from their Snowflake account. How many of those companies found their IT and security departments found the fact that they were using Snowflake in that way a surprise or that they didn’t have multi factor authentication turned on?
How many times was that a surprise? And I think it’s going to be an unfortunately high number. Yes,
Andrew: And what I unfortunately foresee happening is executives will just say, fine, security, IT, go fix it. Without allocating enough resources.
Jerry: yes you failed in your job, which, I guess it’s not a it’s not a completely unfair statement. But on the other hand, I think we have to be we have to be enabled in our jobs. And I’m not sure that always happens.
Andrew: Yeah. Yeah.
Jerry: So anyway, That is that is the stories for today. Oh, go ahead. Sorry. I think there’s a lag now.
Andrew: Yeah. We might be,
Jerry: I
Andrew: stepped each other a little bit this this show, but we’ll figure it out.
Jerry: think there’s a I don’t know if it’s because of the way we’re recording or what, but I think there’s a bit of a lag. So in any event that is the show for today. I appreciate everybody’s attention and hope you found this useful. And if you like the show, you can. Go listen to back episodes.
Everything is available on our website at www. defensivesecurity. org or on your favorite podcast player. If you do the show, we would we would love, love, love for you to give us a five star review that that helps make sure that other people are able to find us and and get us.
And by the way, if you don’t like the show. You can also still give us a five star review.
Andrew: Even more
Jerry: And just not listen. Yeah.
Andrew: That’s true.
That it’s free five stars is free
Jerry: Yeah, and then you just don’t listen anymore.
Anyway.
Andrew: just play it for your cats. That’s what I do.
Jerry: Yes.
I’m not even going to continue down that road.
Andrew: That’s fair, it’s going off the rails.
Jerry: All right. You can find a Mr. Kellett. On the social media where
Andrew: I’m on Twitter slash X at atlerg L E R G and on InfoSec Exchange on the Fediverse, also atlerg L E R G.
Jerry: Awesome you can find me on the fediverse at jerry at infosec. exchange and with that We will talk to you again very soon. Thank you all. Have a great week
Andrew: week. Bye bye.
The AI-powered Podcast Player
