Cyber Security & Cloud Podcast CSCP S03E00 - Steve Wilson - Log4J and Log4Shell Special - Nightmare On Christmas Eve
CSCP is Coming back with Season 3 in the new year!
As a teaser, we bring you the latest story on the blog...Log4j with Steve Wilson from Contrast Security
Steve Wilson is an Application Security expert development manager and currently and currently the head of product at Contrast. Steve joins the podcast to discuss the nightmare just unleashed, log4j, that has been affecting everyone around the cybersecurity industry and the reason why we are facing this other pandemic
We will return with a special launch in 2022 with some special guest
The episode is brought you by AppSec Phoenix Ltd with the Phoenix platform you can make Vulnerability management for software and organization SMART. Follow the tag #appsecsmart
https://www.appsecphoenix.com get a free 30-day licence quoting CSCP https://landing.appsecphoenix.com/register
0:28 Introducing Steve
2:13 Cybersecurity Advice
3:15 Supply chain issues
8:30 Lg4J
12:47 Issue of Supply and software
19:16 What to do to avoid
23:07 Why we are getting it wrong
27:52 Final Positive Message
29:40 Outro
Steve Wilson
Twitter @virtualsteve
https://www.linkedin.com/in/wilsonsd/
Cyber Security and Cloud Podcast hosted by Francesco Cipollone
Twitter @FrankSEC42
#CSCP #cybermentoringmonday cybercloudpodcast.com
Social Media Links
Follow us on social media to get the latest episodes:
Website: http://www.cybercloudpodcast.com/
You can listen to this podcast on your favourite player:
Itunes: https://podcasts.apple.com/gb/podcast/the-cyber-security-cloud-podcast-cscp/id1516316463
Spotify: https://open.spotify.com/show/3fg8AqP4vEi5Im8YKxazUQ
Linkedin: https://www.linkedin.com/company/35703565/admin/
Twitter: https://twitter.com/podcast_cyber
Youtube https://www.youtube.com/channel/UCVgsq-vMzq4sxObVonDsIAg/
Full Transcript
00:00.00
franksec
Hello everyone and welcome back to another episode of the cyber security and cloud podcast today. We have a topic that probably nobody has ever spoken in the recent time that is Goingnna be obligation security vulnerability management but the whole thing that has taken. By the storm the industry that is fundamental log for js and today we have a special guest but before we crack on. Let let us start with our intro.
00:54.11
franksec
All right? or right or right we are Back. So I'd like to welcome steel wilson that came we started chatting over over a Twitter over Twitter threadad around of course up for j. So I've reminded him on the show to actually chat a little bit about the topic and his particular take is been He's the chief product officer of contra security 1 product that we absolutely love and we saw that was quite well reacting on the log four j issue but also he is an early member of the Java team on the early ninety s. But before I talk through it. Let me welcome steve steve welcome on the show.
01:33.74
Steve Wilson
Hey thank you Francisco for having me really looking forward to it. So.
01:37.60
franksec
Brilliant and can you give our audience a little bit about your background. What brought you into side by you know how did you start the journey from the early days with java.
01:47.24
Steve Wilson
Yeah, so um, I started out really early in my career back in the ninety s at Sun microsystems I was an early member of the Java development team. Um. Went on from working really around development tools developer tools for several years and then shifted my focus over to cloud and I spent a lot of time at large companies like oracle and citrix building cloud services and cloud infrastructure and really got exposed. To a lot of the security challenges that are out there in the industry and decided about a year ago that I wanted to really move into the cyber security industry from the inside and so I joined contrast a little over a year ago to head product development.
02:35.60
franksec
Nice, fantastic. And and we need we need more more ally in Cyber especially over over these challenging time. But we have a tradition on the show that we give an overview on the industry of what's working. What's not working so what will be your take on on.
02:53.16
Steve Wilson
Yeah, so um, with the area of the industry that we're really focused on looking at the security of applications and code. It's a really challenging environment out there I Think what we really see is that.
02:53.40
franksec
What's going on.
03:11.40
Steve Wilson
Over the past several years. The complexity in software out there means that the number of security vulnerabilities in a typical program is is escalating dramatically as they get larger and more complicated and really the fact is human brains have a hard time. Ah, dealing with the complexities in the number of paths and things that are through the code today and so you know really this industry around application security has developed there to create tools that ah people can use to make their applications more secure. But 1 of the big shifts going on now is really moving from a focus on standalone security teams working to audit applications sort of almost after they're done to really bringing that security mindset into development at the beginning. And really creating a new culture where um, security comes very early in the cycle of what's going on with code development.
04:18.55
franksec
Right? And I Ah think I think we move towards that space. But as you rightfully say the number of vulnerability and the number of issues that a lot of organizations are finding are escalating over and over and over. And that's just on application security. But then you know development team and now devops teams are faced with you know the Cloud issue the Cloud misconfiguration the deployment in the Cloud then the container base container Image. You know the landscape is in my opinion becoming quite quite. Ah, intense and it' complicated for developer team and security team to have that broad spectrum of knowledge. But then you take even an executive they need to make decision of what is your target. What? what is security what security looking like or what good looks like.
05:11.36
Steve Wilson
Yeah, well I think that in what I'll call the olden days which were really not that long ago in a Pre-cloud world. You could depend a lot more on the idea that many of your applications were hidden behind a firewall that they were.
05:11.59
franksec
What's your take on that.
05:29.29
Steve Wilson
Not exposed to the internet and thus less valuable in ah in a cloud-based world in a zero trust-based world more and more of your applications really are on the internet and that means that every 1 of these vulnerabilities is a potential place that you could be exploited and.
05:37.96
franksec
Um.
05:47.79
Steve Wilson
You know when we start working with a new customer and help them start to evaluate their applications. We'll find that that typical applications have dozens of vulnerabilities in them potentially serious ones and then you look at ah at a large corporation. They may have thousands of applications.
06:05.84
franksec
Right.
06:07.73
Steve Wilson
In their environment. So it's it's not uncommon to see a fortune five hundred or global 2000 company having tens of thousands of discrete vulnerabilities in their software and so from an executive point of view. The question is how do you manage that there's. Ah, sometimes a snap back reaction that says we better stop everything that we're going to that we're doing and and fix this on the other hand. Every company today is a software company. Your competitive advantage is in your software your ability to compete in the market your ability to deliver new services is dependent on that and so the challenge as a leader is how do I balance the real risk.
06:36.69
franksec
Right.
06:51.50
Steve Wilson
With my my need to compete in the market and deliver new value to my customers.
06:55.30
franksec
Right? And you know I like your take I Really like your take on the rest because I think um because there're a lot of tooling around different areas. You know you have Cloud Security Infrastructure security container Security. You know you have your pantasy rapport coming in your read teaming just trying in different things. Your ah security lifecycle tooling that is dust must and you name me rast you know and and and more ah more of those coming and despite that every tool is is doing.
07:21.36
Steve Wilson
So.
07:29.34
franksec
A different level of of ah scanning and and trying to reduce the false positive I think what we're missing in a lot of program of work and a lot of these organization is the contextualization and and the Breadth of view of ah where are those kind of element deployed. That could potentially ah in my in my humble opinion simplify a lot of those kind of conversations and the conversation that traditionally happened between security team development team and executive because everybody could have an opinion on that while. If we display the complexity of the landscape nobody will be able to inform the opinion unless they're very technical. So. What do you think? steve.
08:12.42
Steve Wilson
Yeah, so this this element of risk analysis is is really critical and you know log for J is a really good example of this This is this is an exploit or ah, a vulnerability that has exploits that are incredibly high risk. Right? It's ah it's a 10 out of 10 Cvs Cvss score because it's you know you're you're basically enabling complete remote code execution on your servers and it's really easy to exploit. But when you really go look at it and.
08:32.60
franksec
So.
08:46.86
Steve Wilson
And we've been looking at this specifically with customers. You know we estimate something like fifty fifty six percent of the Java applications out there are packaging of vulnerable um version of log for j but when you really look at it. It actually matters how you use it? um.
08:55.91
franksec
Right.
09:06.14
Steve Wilson
Whether your application is vulnerable and so being able to have tools that are able to analyze. Not just do you have 1 of these things the sort of Naive view. But but are you really vulnerable. That's really really critical to you being able to. For example, prioritize the work that you're going to do? What are you going to mitigate first because again, if you have thousands of applications. You know how are you going to do this all at once can't can't do this in a day this is going to be going on honestly for weeks or months. Um, so yeah, being able to really.
09:30.32
franksec
Where is still not right.
09:41.79
Steve Wilson
Establish risk in an urgent situation like this for triage but then more on a day-to-day basis when you're dealing with an environment where um, you know dozens hundreds or even thousands of software developers continually building New software. How do you evaluate the the risk of different. Um, Conditions vulnerabilities and really decide where you need to make compromises in terms of your development and and really lean into to securing yourself versus continuing to generate that that new business value.
10:15.40
franksec
Right? up. Absolutely agree and and I think the other thing that we saw that that was working was also trying to prioritize the things that are externally exposed that is easily attackable and you know every team right now is scrambling and trying to find a way to. As you rightfully say you know if you if you belong to an enterprise that has multiple deployment even your web come could be bulletproof to log for j but maybe if we take a step back? Um I wanted to understand considering you come from that kind of environment in Java in the early days I want to understand. What happened in there. Why why are we facing with ah vulnerability that is so easy to exploit that should be really never been in the place you know something so trivial ascend a string and that string can then execute. Ah whatever rce or remote code execution. And then download whatever payload you can want and want how how are we in that situation in the year twenty twenty twelve 2.
11:19.86
Steve Wilson
So um, it's it's really interesting to think back to the early days of java and so much emphasis was on creating it as a secure environment. You know, really Java pioneered these concepts like having the the security manager in the runtime that managed what permissions.
11:29.22
franksec
Right.
11:39.81
Steve Wilson
Things had but but a lot of that in in the inception of Java was you have to rewind so far to remember that Java was originally intended for environments like set top boxes and running applets in a browser and so the the security manager was for things like making sure that um.
11:50.79
franksec
And.
11:58.32
Steve Wilson
your your java applet couldn't escape the sandbox and get onto somebody's desktop um the actual security of getting something into the Java runtime environment wasn't what the team was optimizing for originally and so when when you look at this log for j. Vulnerability I think there's a couple of things that come in obviously logging is in some ways the least glamorous thing you know task that you can think of and um, you know that log this log for j library is more than 20 years old it's been
12:25.45
franksec
Rise.
12:35.84
Steve Wilson
You know it got created then it got donated to apache. It's been in Apache for 20 years now with ah with a very small team of honestly very dedicated folks maintaining it but but it's ah it's a small team with minimal investment and minimal tooling. And while it doesn't seem glamorous. Um, this library has been copied literally millions of times different versions of it at different points in different physical locations. So you know you think about? Okay there's a bug and I want to patch the bug. All right? Well, that's that's 1 challenge but the problem is the the offending code has been copied millions of times around the planet. So. There's there's no single place to fix it on top of that. Um, you know the the.
13:17.52
franksec
Drive.
13:26.43
Steve Wilson
Confluence of events that create this vulnerability and make it exploitable are pretty insidious in terms of the the snarly code path you have to go through and while the exploit is trivial. Um, the vulnerability is actually really intricate and so you know what that means is the. The first attempt that the team put out at apache to fix the vulnerability. Um it. It didn't even fix it so you know people went out and started patching to a new version of the log for j library and now they're having to go back and do it again and so in in a lot of ways I think what we're going to find is. Is people continuing to hammer on some of this and until we really get to the bottom of it and then we're going to start the long arduous process of patching this um and we have you know.
14:16.18
franksec
Um, at scale.
14:19.75
Steve Wilson
Certain places where they have tooling in place and they're able to execute very very quickly on it and that's you know 1 of the things we're really proud about at contrast is that I think we have tooling that in some ways was designed for the fact that someday this would happen and and it's been great to work with. Customers and and kind of feel like we're helping them. But so many places don't have that kind of tooling in place they're using. Um you know, free and open source tools to do their software composition analysis that don't have enterprise level management. They're writing scripts trying to figure this out themselves. And then you get all the way to the limit case you know you mentioned something like your webcam could be vulnerable and that's not absurd at all. We've seen out in the industry now very specific attacks where people are targeting things like s and mp where they're actually going out and looking for embedded devices.
15:00.21
franksec
Yeah.
15:13.72
Steve Wilson
And those embedded devices are going to have in some cases literally no way to update them.
15:19.39
franksec
Right? And you know I want to cover this in detail. But before we jump on that we have to we had to have a small section for our sponsors so bear with me a second.
16:16.36
franksec
All right bra and and thank you again for up Phoenix or our sponsor and and keeping us running but I wanted to to touch point on this on this particular topic because I remember Jeff ah kind of wrote a white paper like. 6 or 7 years ago and it actually presented it to black cat as well. This is not a new thing. The industry has been screamed about this is something that will happen. This is something that will be out there and and now it suddenly happened and I ah do also subscribe to your view and. To your pain in a way that code has been forked so many times and have been distributed in so many places that it becomes very very complex to fix it and we're never going to know that the the extreme expansion but maybe on on on there the more scary topics that I want. As to maybe debate if that's what was 1 library. What's stopping attacking now or poking at the other side of libraries to discover um, similar log for j kind of problems. What do you think.
17:25.19
Steve Wilson
Well look the the way I'd like to say this is this has happened before and it will happen again right? if we if we rewind a few years ago to 2017 the apache struts library had a severe vulnerability in it and that is um.
17:30.97
franksec
Um.
17:38.57
franksec
Right.
17:44.91
Steve Wilson
Ah, a less used library than log for J but the same basic concept is there popular open source library embedded in lots and lots of places with a vulnerability in it that could lead to really severe consequences and. You know what's interesting is the world remembers this vulnerability but they don't remember it as the strut's vulnerability. They remember it as the Equifax breach right? and there were many people that were breached from that. But if you don't remember this 1 about 1 hundred and fifty million people lost.
18:08.30
franksec
Ah, right.
18:20.75
Steve Wilson
Their their personal financial info from equifax which is 1 of the global credit rating organizations and as a result they they wound up paying four hundred and 25 million dollars in fines for not being secure. Um, but the the interesting thing here is. Um, did the world learn anything from this and they absolutely did right? if you look at the difference in response between the Struts vulnerability and the log for j vulnerability um, 1 of the reasons that Equifax was penalized so heavily. Is they could have done much better. This was for them. Not a zero day vulnerability. It was a disclosed vulnerability. It was well known. There were patches that were available and they simply did not act on it. Um.
19:01.11
franksec
Um, is a well known.
19:16.79
Steve Wilson
What's interesting here to see the difference. 4 years later is that the industry realized how serious this was um, you know I yeah yeah you know on thursday night last week people started.
19:23.25
franksec
Um, enacted fast.
19:33.61
Steve Wilson
Exploiting this in minecraft of all places you know minecraft the popular video game. Um, you know famously is written in Java you know I remember a few years ago my daughter went to coding camp over the summer and learns to write her first java programs as Minecraft extensions. So you know. Probably millions of people learned to program by hacking on minecraft and so um, in some ways. It's it's not surprising that that was the not the first place that this was exploited but the the place people realized how serious this was is people were exploiting this by.
19:56.27
franksec
Um, has great.
20:05.97
franksec
Right.
20:10.39
Steve Wilson
Putting messages into the minecraft chat window that was how easy it was to exploit. Um, but that was happening on Thursday and thursday night you know our research team at contrast started getting information about this. Um, you know I heard something about it and I went to bed and I got up at. 5 in the morning the next morning I get up early I'm on the west coast of the us and we have teams in europe so I get up early to talk to them and I had slack messages from our our chief architect that said stevie need to call me right now and I talked to him and he said you know by Friday morning he said.
20:42.78
franksec
Um.
20:49.10
Steve Wilson
Steve this is the most serious thing I've ever seen. We have to help our customers get in front of this and so you know you started to see the news coming out on Friday people were reacting to it not everywhere. There's it's it's far from perfect and it's.
21:02.89
franksec
It was pocket.
21:06.36
Steve Wilson
Far from uniform but but the industry is jumping on this and there are let's say the more advanced shops are much better prepared. The tooling is better. It's absolutely better than it was 4 years ago and so we we have moved forward from that. But then your question is will this happen again. Of course it will um the the fact that we still build software where you know you see different different figures but up to 80 percent of the code in a typical business application is open source.
21:26.94
franksec
Nope yeah.
21:40.45
Steve Wilson
And so really, what people are starting to talk about you know, started before this really going back to solar winds. But the the topic around software supply chain management is now the hot topic and I think that's actually a really good way to phrase it because it makes it a bigger problem than just.
21:52.50
franksec
And right.
21:59.78
Steve Wilson
Thinking about managing vulnerabilities. It's about understanding where your codes coming from what's the Providence of it and being able to really understand that end to end and I think that's going to be the next step in making this better.
22:12.55
franksec
So show. Will we start seeing vul be deploying stock trace. That's gonna be the next 1 gonna get it. Ah am I giving wrong suggestion of the wrong people. Ah.
22:18.64
Steve Wilson
Oh my? yeah.
22:28.90
franksec
Ah, you know because after after open source destins used kind of to by every single developer on earth and I'm pretty actually some of my friends actually have done this experiment of publishing exploit and poc with vulnerable code in there so you had hackers actually just blindfoldingly. Trusting a piece of software just downloading executing it with boom in there and and a callback home and it was a friendly experiment by Andy hilllabs. But um, it was quite interesting to see how blind trust was deployed on. You know piece of code running on the web that is like going outside and asking candy to a strangerr right.
23:12.17
Steve Wilson
Yeah, well the um, you know the the more insidious example of this is something we started to see earlier. This year is a rise in um, a tax that it's going by different names but dependency confusion is 1 of them.
23:29.10
franksec
The.
23:31.89
Steve Wilson
And when you think about the way that that people's build systems and cicd systems work they're they're constantly going out on the internet and pulling down these packages from massive open source repositories where you actually you know you're you're somewhat hoping that you're getting the right thing. And actually a lot of the ways that these work you're you're only providing a general description of the package that you want and it's trying to find the 1 that's best fit and people have found that they can go and create their own version of popular open source libraries put them up in those repos and have people pull them down and um. 1 of our researchers at contrast went went did a proof of concept with this went and looked for applications that looked like they were exposed to this and actually Microsoft teams wound up being a good example now Microsoft's an investor and a partner. Um. Ah, and we're in their bug bounty Program. So we we did this all above board but we actually created some open source libraries and Microsoft pulled them down and compiled them into into their binary and it was just an example.
24:40.97
franksec
Teams.
24:45.22
Steve Wilson
Of How even a sophisticated software shop um can be vulnerable to this so you know they've hardened their processes since then but other people have not This is a really new example of ah of a vulnerability out there being able to divert the software supply chain. Um. To you know a Hacker's nefarious ends and so the ability of someone to go and create their own version of an open source library with some nefarious code. You know we've seen this so far largely people doing things like dumping in crypto minorers and and that's well documented. But.
25:21.90
franksec
Bri yeah or run somewhere. That's I think I saw I saw a couple of days ago. Ah, payload and conti starting to deploy this as as potentially run some arrow or or run some my payload so we start seeing.
25:24.20
Steve Wilson
We know there must be examples of much more defarious usage. Absolutely.
25:41.22
franksec
Fundamentally ransome are going towards this and that's that's the other scary part that the industry from the Attacker prospect. This seems to have industrialized the use of this massive scale vulnerability and decimal scary factor that we had just a week or maybe 2 time to actually breathe text vulnerabilit be so time to detection and and and remediation is actually being shorted dramatically I mean our ourtistic goes from roughly 3 to fifteen days to deploy something like this at scale and it's being confirmed basically by this but it's. Think is is a scary factor and then on the other side maybe here more in the u k we saw fundamentally british airways being attacked with a much more malicious code where somebody ah fundamentally hijacked 1 of the developer trusted account and. Injected malicious code e in a library so that's that's even worse you know and I agree with you. It's it double down on the subject of controlling your supply chain but controlling how you pull in things where you're deploying and. In my humble opinion I think we've been. We've been using security in the wrong way right now and we've being putting them in the front foot and firefighting vulnerability on day in and the out and they kind of lost their way by not focusing on systemating and on strategic thing like creating. Ah, proxy for libraries or or analyzing open source of what comes in and out like what the the security team in contrast does and that's how we should be using back security for that instrumental systemic change rather than day in and out management of vulnerability.
27:26.62
Steve Wilson
So yeah I mean look I think the the day-to-day management of vulnerability actually to some extent hasn't been done at all in a lot of shops right? It's been um, it's been completely pushed off to a.
27:26.89
franksec
What do you think safe.
27:36.95
franksec
Ah.
27:43.92
Steve Wilson
Ah, periodic scanning based procedure run by the security team where you scan things on a quarterly or even yearly basis and I lived this in my last job it's 1 of the reasons I got excited about about this job opportunity when it came up was I was running a large development team. And the head of engineering came to me and said I need to cancel all the features that I promised for next quarter because the security team just ran a scan and filed a thousand jira tickets. Um, and and now there was this record of this potential vulnerability that we were obliged to deal with and it turned out. Most of them weren't real vulnerabilities almost all of them weren't um, but it wound up being a huge amount of work to so to sift through it on the other hand for for companies that really adopt this devsec ops attitude and get the right tooling in place to enable it. Um, you find a potential vulnerability maybe before you even complete your pull request to put the put the software back and it's just like any other bug if the bug gets into the code base. It's 10 times as expensive to fix it as it was for the developer to fix it on their desktop. Um, if it actually gets out to a customer It's a Hundred times more expensive and you know with security given the stakes. It's much worse than that. So um, the the real shift here is to push so much more of the responsibility down to this. To the developers but also really not make the developers responsible for it because it's hard for developers but to put the right tool chain around them that makes it easy and it really is possible with the modern tools to do that now and that's the big opportunity to change how we do development.
29:35.39
franksec
Brian I agree with you. It should be It should be a collaboration between shift left and the copy is on more automation in the place because a lot of this as you rightfully say is still pretty much reactive is still pretty much that debate in Discussion. And then the endless argument between the se security team and the development team saying this is false positive. This is internal is a false positive rather than you know it's accept the risk and is different priorities and stuff like that. So. I think we can do better at thefsecops to actually remove security people on doing consistently these firefighting in this endless debate. Um, and and and automate a lot of the relationship but also the detection of um false positive based on contextual aspect and contextual information. If you can actually exploit it if it's actually visible to attack. Ah then you know we we focus on it because otherwise we're going to be always overflloded by these issues and you know look for js all similar are going to keep on piling up right.
30:42.60
Steve Wilson
Absolutely I mean I think we really do have the the tools at our disposal and the processes being developed out there in the industry to to just fundamentally shift this change the game and make this so much more efficient and create. Really much more secure applications as a result. So.
31:00.59
franksec
Fantastic! and I guess we we this is just a a nice input to the to the conclusion that is the positive message on our industry. So if you want to double down on that Steve what will be your positive message overall rather than we. We have the 2 and we have the technology and we can rebuild this. Ah.
31:20.62
Steve Wilson
Like I think going going back to a little bit earlier I think the good news is you know this has happened before the industry has moved a tremendous distance since the Struts vulnerability for example, um, this really would be much worse. If we weren't in the position that we are now that we had better understanding of the risks better tools better processes. We have the tools out there now widely deployed to understand your your open source footprint. What's vulnerable. Um, we have the tools in place that help people upgrade and fix this. We even have tools today like like rasp tools that can protect you and we've seen evidence that these rash tools were protecting people um before day zero now. So really, we're in a position where we're moving forward.
32:09.23
franksec
Um.
32:15.56
Steve Wilson
So quickly that look there's no end in sight for this but really, the bar has raised dramatically and if we work together as an industry the next time this happens we'll be even better prepared.
32:27.90
franksec
Fantastic. And yeah I agree with you. We've seen an enormous collaboration between teams and information out there. So I Really appreciated that collaboration and and enjoy that seeing that collaboration and the community getting together to to fix. But ah on the conclusion of the show if people want to find more about what you do day in in day out where where is the best place for them to contact you and how they can reach you yet. Stay.
32:53.99
Steve Wilson
Yeah, so please so please come over check out what we're doing at the Contrastsecurity Dot Com Website. You can get all the details on all of our commercial tools. Also check out our blog there. There's a link off the front page to some free and open source tools that we've put out to help with log for J in particular so we really want people in the community to engage with us on this also feel free to reach out to me direct on linkedin.
33:23.13
franksec
All right brave and everybody. Thank you very much we we understand that everybody is tired and stressed. We really hope that everybody can enjoy christmas at some stage or time and get away from the lock for j unfortunately attack it don't sleep so defend it on. Don't sleep either. But we're gonna get ahead of this together. So this is your host francesco I had the pleasure to talk with Steve wilson the chief product officer for contra security and I wish you everybody to stay safe and have a lovely christmas Thank you.
00:00.00
franksec
Hello everyone and welcome back to another episode of the cyber security and cloud podcast today. We have a topic that probably nobody has ever spoken in the recent time that is Goingnna be obligation security vulnerability management but the whole thing that has taken. By the storm the industry that is fundamental log for js and today we have a special guest but before we crack on. Let let us start with our intro.
00:54.11
franksec
All right? or right or right we are Back. So I'd like to welcome steel wilson that came we started chatting over over a Twitter over Twitter threadad around of course up for j. So I've reminded him on the show to actually chat a little bit about the topic and his particular take is been He's the chief product officer of contra security 1 product that we absolutely love and we saw that was quite well reacting on the log four j issue but also he is an early member of the Java team on the early ninety s. But before I talk through it. Let me welcome steve steve welcome on the show.
01:33.74
Steve Wilson
Hey thank you Francisco for having me really looking forward to it. So.
01:37.60
franksec
Brilliant and can you give our audience a little bit about your background. What brought you into side by you know how did you start the journey from the early days with java.
01:47.24
Steve Wilson
Yeah, so um, I started out really early in my career back in the ninety s at Sun microsystems I was an early member of the Java development team. Um. Went on from working really around development tools developer tools for several years and then shifted my focus over to cloud and I spent a lot of time at large companies like oracle and citrix building cloud services and cloud infrastructure and really got exposed. To a lot of the security challenges that are out there in the industry and decided about a year ago that I wanted to really move into the cyber security industry from the inside and so I joined contrast a little over a year ago to head product development.
02:35.60
franksec
Nice, fantastic. And and we need we need more more ally in Cyber especially over over these challenging time. But we have a tradition on the show that we give an overview on the industry of what's working. What's not working so what will be your take on on.
02:53.16
Steve Wilson
Yeah, so um, with the area of the industry that we're really focused on looking at the security of applications and code. It's a really challenging environment out there I Think what we really see is that.
02:53.40
franksec
What's going on.
03:11.40
Steve Wilson
Over the past several years. The complexity in software out there means that the number of security vulnerabilities in a typical program is is escalating dramatically as they get larger and more complicated and really the fact is human brains have a hard time. Ah, dealing with the complexities in the number of paths and things that are through the code today and so you know really this industry around application security has developed there to create tools that ah people can use to make their applications more secure. But 1 of the big shifts going on now is really moving from a focus on standalone security teams working to audit applications sort of almost after they're done to really bringing that security mindset into development at the beginning. And really creating a new culture where um, security comes very early in the cycle of what's going on with code development.
04:18.55
franksec
Right? And I Ah think I think we move towards that space. But as you rightfully say the number of vulnerability and the number of issues that a lot of organizations are finding are escalating over and over and over. And that's just on application security. But then you know development team and now devops teams are faced with you know the Cloud issue the Cloud misconfiguration the deployment in the Cloud then the container base container Image. You know the landscape is in my opinion becoming quite quite. Ah, intense and it' complicated for developer team and security team to have that broad spectrum of knowledge. But then you take even an executive they need to make decision of what is your target. What? what is security what security looking like or what good looks like.
05:11.36
Steve Wilson
Yeah, well I think that in what I'll call the olden days which were really not that long ago in a Pre-cloud world. You could depend a lot more on the idea that many of your applications were hidden behind a firewall that they were.
05:11.59
franksec
What's your take on that.
05:29.29
Steve Wilson
Not exposed to the internet and thus less valuable in ah in a cloud-based world in a zero trust-based world more and more of your applications really are on the internet and that means that every 1 of these vulnerabilities is a potential place that you could be exploited and.
05:37.96
franksec
Um.
05:47.79
Steve Wilson
You know when we start working with a new customer and help them start to evaluate their applications. We'll find that that typical applications have dozens of vulnerabilities in them potentially serious ones and then you look at ah at a large corporation. They may have thousands of applications.
06:05.84
franksec
Right.
06:07.73
Steve Wilson
In their environment. So it's it's not uncommon to see a fortune five hundred or global 2000 company having tens of thousands of discrete vulnerabilities in their software and so from an executive point of view. The question is how do you manage that there's. Ah, sometimes a snap back reaction that says we better stop everything that we're going to that we're doing and and fix this on the other hand. Every company today is a software company. Your competitive advantage is in your software your ability to compete in the market your ability to deliver new services is dependent on that and so the challenge as a leader is how do I balance the real risk.
06:36.69
franksec
Right.
06:51.50
Steve Wilson
With my my need to compete in the market and deliver new value to my customers.
06:55.30
franksec
Right? And you know I like your take I Really like your take on the rest because I think um because there're a lot of tooling around different areas. You know you have Cloud Security Infrastructure security container Security. You know you have your pantasy rapport coming in your read teaming just trying in different things. Your ah security lifecycle tooling that is dust must and you name me rast you know and and and more ah more of those coming and despite that every tool is is doing.
07:21.36
Steve Wilson
So.
07:29.34
franksec
A different level of of ah scanning and and trying to reduce the false positive I think what we're missing in a lot of program of work and a lot of these organization is the contextualization and and the Breadth of view of ah where are those kind of element deployed. That could potentially ah in my in my humble opinion simplify a lot of those kind of conversations and the conversation that traditionally happened between security team development team and executive because everybody could have an opinion on that while. If we display the complexity of the landscape nobody will be able to inform the opinion unless they're very technical. So. What do you think? steve.
08:12.42
Steve Wilson
Yeah, so this this element of risk analysis is is really critical and you know log for J is a really good example of this This is this is an exploit or ah, a vulnerability that has exploits that are incredibly high risk. Right? It's ah it's a 10 out of 10 Cvs Cvss score because it's you know you're you're basically enabling complete remote code execution on your servers and it's really easy to exploit. But when you really go look at it and.
08:32.60
franksec
So.
08:46.86
Steve Wilson
And we've been looking at this specifically with customers. You know we estimate something like fifty fifty six percent of the Java applications out there are packaging of vulnerable um version of log for j but when you really look at it. It actually matters how you use it? um.
08:55.91
franksec
Right.
09:06.14
Steve Wilson
Whether your application is vulnerable and so being able to have tools that are able to analyze. Not just do you have 1 of these things the sort of Naive view. But but are you really vulnerable. That's really really critical to you being able to. For example, prioritize the work that you're going to do? What are you going to mitigate first because again, if you have thousands of applications. You know how are you going to do this all at once can't can't do this in a day this is going to be going on honestly for weeks or months. Um, so yeah, being able to really.
09:30.32
franksec
Where is still not right.
09:41.79
Steve Wilson
Establish risk in an urgent situation like this for triage but then more on a day-to-day basis when you're dealing with an environment where um, you know dozens hundreds or even thousands of software developers continually building New software. How do you evaluate the the risk of different. Um, Conditions vulnerabilities and really decide where you need to make compromises in terms of your development and and really lean into to securing yourself versus continuing to generate that that new business value.
10:15.40
franksec
Right? up. Absolutely agree and and I think the other thing that we saw that that was working was also trying to prioritize the things that are externally exposed that is easily attackable and you know every team right now is scrambling and trying to find a way to. As you rightfully say you know if you if you belong to an enterprise that has multiple deployment even your web come could be bulletproof to log for j but maybe if we take a step back? Um I wanted to understand considering you come from that kind of environment in Java in the early days I want to understand. What happened in there. Why why are we facing with ah vulnerability that is so easy to exploit that should be really never been in the place you know something so trivial ascend a string and that string can then execute. Ah whatever rce or remote code execution. And then download whatever payload you can want and want how how are we in that situation in the year twenty twenty twelve 2.
11:19.86
Steve Wilson
So um, it's it's really interesting to think back to the early days of java and so much emphasis was on creating it as a secure environment. You know, really Java pioneered these concepts like having the the security manager in the runtime that managed what permissions.
11:29.22
franksec
Right.
11:39.81
Steve Wilson
Things had but but a lot of that in in the inception of Java was you have to rewind so far to remember that Java was originally intended for environments like set top boxes and running applets in a browser and so the the security manager was for things like making sure that um.
11:50.79
franksec
And.
11:58.32
Steve Wilson
your your java applet couldn't escape the sandbox and get onto somebody's desktop um the actual security of getting something into the Java runtime environment wasn't what the team was optimizing for originally and so when when you look at this log for j. Vulnerability I think there's a couple of things that come in obviously logging is in some ways the least glamorous thing you know task that you can think of and um, you know that log this log for j library is more than 20 years old it's been
12:25.45
franksec
Rise.
12:35.84
Steve Wilson
You know it got created then it got donated to apache. It's been in Apache for 20 years now with ah with a very small team of honestly very dedicated folks maintaining it but but it's ah it's a small team with minimal investment and minimal tooling. And while it doesn't seem glamorous. Um, this library has been copied literally millions of times different versions of it at different points in different physical locations. So you know you think about? Okay there's a bug and I want to patch the bug. All right? Well, that's that's 1 challenge but the problem is the the offending code has been copied millions of times around the planet. So. There's there's no single place to fix it on top of that. Um, you know the the.
13:17.52
franksec
Drive.
13:26.43
Steve Wilson
Confluence of events that create this vulnerability and make it exploitable are pretty insidious in terms of the the snarly code path you have to go through and while the exploit is trivial. Um, the vulnerability is actually really intricate and so you know what that means is the. The first attempt that the team put out at apache to fix the vulnerability. Um it. It didn't even fix it so you know people went out and started patching to a new version of the log for j library and now they're having to go back and do it again and so in in a lot of ways I think what we're going to find is. Is people continuing to hammer on some of this and until we really get to the bottom of it and then we're going to start the long arduous process of patching this um and we have you know.
14:16.18
franksec
Um, at scale.
14:19.75
Steve Wilson
Certain places where they have tooling in place and they're able to execute very very quickly on it and that's you know 1 of the things we're really proud about at contrast is that I think we have tooling that in some ways was designed for the fact that someday this would happen and and it's been great to work with. Customers and and kind of feel like we're helping them. But so many places don't have that kind of tooling in place they're using. Um you know, free and open source tools to do their software composition analysis that don't have enterprise level management. They're writing scripts trying to figure this out themselves. And then you get all the way to the limit case you know you mentioned something like your webcam could be vulnerable and that's not absurd at all. We've seen out in the industry now very specific attacks where people are targeting things like s and mp where they're actually going out and looking for embedded devices.
15:00.21
franksec
Yeah.
15:13.72
Steve Wilson
And those embedded devices are going to have in some cases literally no way to update them.
15:19.39
franksec
Right? And you know I want to cover this in detail. But before we jump on that we have to we had to have a small section for our sponsors so bear with me a second.
16:16.36
franksec
All right bra and and thank you again for up Phoenix or our sponsor and and keeping us running but I wanted to to touch point on this on this particular topic because I remember Jeff ah kind of wrote a white paper like. 6 or 7 years ago and it actually presented it to black cat as well. This is not a new thing. The industry has been screamed about this is something that will happen. This is something that will be out there and and now it suddenly happened and I ah do also subscribe to your view and. To your pain in a way that code has been forked so many times and have been distributed in so many places that it becomes very very complex to fix it and we're never going to know that the the extreme expansion but maybe on on on there the more scary topics that I want. As to maybe debate if that's what was 1 library. What's stopping attacking now or poking at the other side of libraries to discover um, similar log for j kind of problems. What do you think.
17:25.19
Steve Wilson
Well look the the way I'd like to say this is this has happened before and it will happen again right? if we if we rewind a few years ago to 2017 the apache struts library had a severe vulnerability in it and that is um.
17:30.97
franksec
Um.
17:38.57
franksec
Right.
17:44.91
Steve Wilson
Ah, a less used library than log for J but the same basic concept is there popular open source library embedded in lots and lots of places with a vulnerability in it that could lead to really severe consequences and. You know what's interesting is the world remembers this vulnerability but they don't remember it as the strut's vulnerability. They remember it as the Equifax breach right? and there were many people that were breached from that. But if you don't remember this 1 about 1 hundred and fifty million people lost.
18:08.30
franksec
Ah, right.
18:20.75
Steve Wilson
Their their personal financial info from equifax which is 1 of the global credit rating organizations and as a result they they wound up paying four hundred and 25 million dollars in fines for not being secure. Um, but the the interesting thing here is. Um, did the world learn anything from this and they absolutely did right? if you look at the difference in response between the Struts vulnerability and the log for j vulnerability um, 1 of the reasons that Equifax was penalized so heavily. Is they could have done much better. This was for them. Not a zero day vulnerability. It was a disclosed vulnerability. It was well known. There were patches that were available and they simply did not act on it. Um.
19:01.11
franksec
Um, is a well known.
19:16.79
Steve Wilson
What's interesting here to see the difference. 4 years later is that the industry realized how serious this was um, you know I yeah yeah you know on thursday night last week people started.
19:23.25
franksec
Um, enacted fast.
19:33.61
Steve Wilson
Exploiting this in minecraft of all places you know minecraft the popular video game. Um, you know famously is written in Java you know I remember a few years ago my daughter went to coding camp over the summer and learns to write her first java programs as Minecraft extensions. So you know. Probably millions of people learned to program by hacking on minecraft and so um, in some ways. It's it's not surprising that that was the not the first place that this was exploited but the the place people realized how serious this was is people were exploiting this by.
19:56.27
franksec
Um, has great.
20:05.97
franksec
Right.
20:10.39
Steve Wilson
Putting messages into the minecraft chat window that was how easy it was to exploit. Um, but that was happening on Thursday and thursday night you know our research team at contrast started getting information about this. Um, you know I heard something about it and I went to bed and I got up at. 5 in the morning the next morning I get up early I'm on the west coast of the us and we have teams in europe so I get up early to talk to them and I had slack messages from our our chief architect that said stevie need to call me right now and I talked to him and he said you know by Friday morning he said.
20:42.78
franksec
Um.
20:49.10
Steve Wilson
Steve this is the most serious thing I've ever seen. We have to help our customers get in front of this and so you know you started to see the news coming out on Friday people were reacting to it not everywhere. There's it's it's far from perfect and it's.
21:02.89
franksec
It was pocket.
21:06.36
Steve Wilson
Far from uniform but but the industry is jumping on this and there are let's say the more advanced shops are much better prepared. The tooling is better. It's absolutely better than it was 4 years ago and so we we have moved forward from that. But then your question is will this happen again. Of course it will um the the fact that we still build software where you know you see different different figures but up to 80 percent of the code in a typical business application is open source.
21:26.94
franksec
Nope yeah.
21:40.45
Steve Wilson
And so really, what people are starting to talk about you know, started before this really going back to solar winds. But the the topic around software supply chain management is now the hot topic and I think that's actually a really good way to phrase it because it makes it a bigger problem than just.
21:52.50
franksec
And right.
21:59.78
Steve Wilson
Thinking about managing vulnerabilities. It's about understanding where your codes coming from what's the Providence of it and being able to really understand that end to end and I think that's going to be the next step in making this better.
22:12.55
franksec
So show. Will we start seeing vul be deploying stock trace. That's gonna be the next 1 gonna get it. Ah am I giving wrong suggestion of the wrong people. Ah.
22:18.64
Steve Wilson
Oh my? yeah.
22:28.90
franksec
Ah, you know because after after open source destins used kind of to by every single developer on earth and I'm pretty actually some of my friends actually have done this experiment of publishing exploit and poc with vulnerable code in there so you had hackers actually just blindfoldingly. Trusting a piece of software just downloading executing it with boom in there and and a callback home and it was a friendly experiment by Andy hilllabs. But um, it was quite interesting to see how blind trust was deployed on. You know piece of code running on the web that is like going outside and asking candy to a strangerr right.
23:12.17
Steve Wilson
Yeah, well the um, you know the the more insidious example of this is something we started to see earlier. This year is a rise in um, a tax that it's going by different names but dependency confusion is 1 of them.
23:29.10
franksec
The.
23:31.89
Steve Wilson
And when you think about the way that that people's build systems and cicd systems work they're they're constantly going out on the internet and pulling down these packages from massive open source repositories where you actually you know you're you're somewhat hoping that you're getting the right thing. And actually a lot of the ways that these work you're you're only providing a general description of the package that you want and it's trying to find the 1 that's best fit and people have found that they can go and create their own version of popular open source libraries put them up in those repos and have people pull them down and um. 1 of our researchers at contrast went went did a proof of concept with this went and looked for applications that looked like they were exposed to this and actually Microsoft teams wound up being a good example now Microsoft's an investor and a partner. Um. Ah, and we're in their bug bounty Program. So we we did this all above board but we actually created some open source libraries and Microsoft pulled them down and compiled them into into their binary and it was just an example.
24:40.97
franksec
Teams.
24:45.22
Steve Wilson
Of How even a sophisticated software shop um can be vulnerable to this so you know they've hardened their processes since then but other people have not This is a really new example of ah of a vulnerability out there being able to divert the software supply chain. Um. To you know a Hacker's nefarious ends and so the ability of someone to go and create their own version of an open source library with some nefarious code. You know we've seen this so far largely people doing things like dumping in crypto minorers and and that's well documented. But.
25:21.90
franksec
Bri yeah or run somewhere. That's I think I saw I saw a couple of days ago. Ah, payload and conti starting to deploy this as as potentially run some arrow or or run some my payload so we start seeing.
25:24.20
Steve Wilson
We know there must be examples of much more defarious usage. Absolutely.
25:41.22
franksec
Fundamentally ransome are going towards this and that's that's the other scary part that the industry from the Attacker prospect. This seems to have industrialized the use of this massive scale vulnerability and decimal scary factor that we had just a week or maybe 2 time to actually breathe text vulnerabilit be so time to detection and and and remediation is actually being shorted dramatically I mean our ourtistic goes from roughly 3 to fifteen days to deploy something like this at scale and it's being confirmed basically by this but it's. Think is is a scary factor and then on the other side maybe here more in the u k we saw fundamentally british airways being attacked with a much more malicious code where somebody ah fundamentally hijacked 1 of the developer trusted account and. Injected malicious code e in a library so that's that's even worse you know and I agree with you. It's it double down on the subject of controlling your supply chain but controlling how you pull in things where you're deploying and. In my humble opinion I think we've been. We've been using security in the wrong way right now and we've being putting them in the front foot and firefighting vulnerability on day in and the out and they kind of lost their way by not focusing on systemating and on strategic thing like creating. Ah, proxy for libraries or or analyzing open source of what comes in and out like what the the security team in contrast does and that's how we should be using back security for that instrumental systemic change rather than day in and out management of vulnerability.
27:26.62
Steve Wilson
So yeah I mean look I think the the day-to-day management of vulnerability actually to some extent hasn't been done at all in a lot of shops right? It's been um, it's been completely pushed off to a.
27:26.89
franksec
What do you think safe.
27:36.95
franksec
Ah.
27:43.92
Steve Wilson
Ah, periodic scanning based procedure run by the security team where you scan things on a quarterly or even yearly basis and I lived this in my last job it's 1 of the reasons I got excited about about this job opportunity when it came up was I was running a large development team. And the head of engineering came to me and said I need to cancel all the features that I promised for next quarter because the security team just ran a scan and filed a thousand jira tickets. Um, and and now there was this record of this potential vulnerability that we were obliged to deal with and it turned out. Most of them weren't real vulnerabilities almost all of them weren't um, but it wound up being a huge amount of work to so to sift through it on the other hand for for companies that really adopt this devsec ops attitude and get the right tooling in place to enable it. Um, you find a potential vulnerability maybe before you even complete your pull request to put the put the software back and it's just like any other bug if the bug gets into the code base. It's 10 times as expensive to fix it as it was for the developer to fix it on their desktop. Um, if it actually gets out to a customer It's a Hundred times more expensive and you know with security given the stakes. It's much worse than that. So um, the the real shift here is to push so much more of the responsibility down to this. To the developers but also really not make the developers responsible for it because it's hard for developers but to put the right tool chain around them that makes it easy and it really is possible with the modern tools to do that now and that's the big opportunity to change how we do development.
29:35.39
franksec
Brian I agree with you. It should be It should be a collaboration between shift left and the copy is on more automation in the place because a lot of this as you rightfully say is still pretty much reactive is still pretty much that debate in Discussion. And then the endless argument between the se security team and the development team saying this is false positive. This is internal is a false positive rather than you know it's accept the risk and is different priorities and stuff like that. So. I think we can do better at thefsecops to actually remove security people on doing consistently these firefighting in this endless debate. Um, and and and automate a lot of the relationship but also the detection of um false positive based on contextual aspect and contextual information. If you can actually exploit it if it's actually visible to attack. Ah then you know we we focus on it because otherwise we're going to be always overflloded by these issues and you know look for js all similar are going to keep on piling up right.
30:42.60
Steve Wilson
Absolutely I mean I think we really do have the the tools at our disposal and the processes being developed out there in the industry to to just fundamentally shift this change the game and make this so much more efficient and create. Really much more secure applications as a result. So.
31:00.59
franksec
Fantastic! and I guess we we this is just a a nice input to the to the conclusion that is the positive message on our industry. So if you want to double down on that Steve what will be your positive message overall rather than we. We have the 2 and we have the technology and we can rebuild this. Ah.
31:20.62
Steve Wilson
Like I think going going back to a little bit earlier I think the good news is you know this has happened before the industry has moved a tremendous distance since the Struts vulnerability for example, um, this really would be much worse. If we weren't in the position that we are now that we had better understanding of the risks better tools better processes. We have the tools out there now widely deployed to understand your your open source footprint. What's vulnerable. Um, we have the tools in place that help people upgrade and fix this. We even have tools today like like rasp tools that can protect you and we've seen evidence that these rash tools were protecting people um before day zero now. So really, we're in a position where we're moving forward.
32:09.23
franksec
Um.
32:15.56
Steve Wilson
So quickly that look there's no end in sight for this but really, the bar has raised dramatically and if we work together as an industry the next time this happens we'll be even better prepared.
32:27.90
franksec
Fantastic. And yeah I agree with you. We've seen an enormous collaboration between teams and information out there. So I Really appreciated that collaboration and and enjoy that seeing that collaboration and the community getting together to to fix. But ah on the conclusion of the show if people want to find more about what you do day in in day out where where is the best place for them to contact you and how they can reach you yet. Stay.
32:53.99
Steve Wilson
Yeah, so please so please come over check out what we're doing at the Contrastsecurity Dot Com Website. You can get all the details on all of our commercial tools. Also check out our blog there. There's a link off the front page to some free and open source tools that we've put out to help with log for J in particular so we really want people in the community to engage with us on this also feel free to reach out to me direct on linkedin.
33:23.13
franksec
All right brave and everybody. Thank you very much we we understand that everybody is tired and stressed. We really hope that everybody can enjoy christmas at some stage or time and get away from the lock for j unfortunately attack it don't sleep so defend it on. Don't sleep either. But we're gonna get ahead of this together. So this is your host francesco I had the pleasure to talk with Steve wilson the chief product officer for contra security and I wish you everybody to stay safe and have a lovely christmas Thank you.