Navigating Cloud Infrastructure Challenges

Mario Gerard: Hello, and welcome to the TPM podcast with your host, Mario Gerard. This is part three, the final part of how to run a large-scale program. Our conversation with Rhea, if you have in third part one and two, definitely check that out before you go ahead and listen to this part. I hope you enjoy it. Continue listening.

Talking about that trend. What are the most common pitfalls you see people make, or, or people need to watch out for?

Rhea Frondozo: So, as a breath TPM, one of the things that I know that has happened to not only me, but the TPMS that I manage is when you work on a large-scale program, you’re working with a lot of different functional area owners, and it’s your job to hold them accountable for getting their work done.

But a lot of times when you come in as the TPM and you come in as such a strong lead, they want to be able to rely on you instead to get the work done and for you to solve their problem. The issue with this is when you’re at breath TPM, you have so many different areas that you are managing, that if you were to do the work for everyone, instead of holding them accountable, ultimately you will fail.

And so, it’s really important as a breath TPM, to make sure that you understand your scope, your responsibility, your accountability, and figure out who it is that you need to rely on to do what work and hold them to that.

Mario Gerard: Yeah. And sometimes you don’t have the right people, what I’ve done in those kind of situations and say, Hey, talk to your senior leadership within your organization, and if you want, I will recommend somebody within that larger organization who I think can go ahead and do this for you, but you don’t step in and help fix somebody else’s problem, because then it becomes your problem. And then they kind of walk away.

So, you want your pocs or your functional area owners to kind of own their space and to work on the problem and then get back to you on the milestones on how they’re doing on it rather than you are running those smaller programs.

Rhea Frondozo: Right. And this is where that judgment call is really necessary. Like how much you step in to help them get them on the right track versus you continue ensuring that they keep on track versus you doing it yourself.

Mario Gerard: And where you step into help sometimes. Because sometimes they don’t have a very strong lead and then you might go in to reactivate that program or put it on the right track and then ensure that you’re monitoring it to some degree, but you’re not actually going and doing all the work.

Rhea Frondozo: And this is where trying to figure out kind of that line between how much you go in and try and help yourself versus how much you invest instead in making an escalation to leadership to ask for the help that you need.

And so again, this comes down to a judgment call of where you spend your time as a TPM to make sure that your program as a whole is successful.

Mario Gerard: Yeah. I think, I think one of the key things which I’ve learned, I am working at OCI was to always reevaluate where I’m spending my time. This is on literally on a daily basis or on a weekly basis. Like where am I spending most of my time? And is it the right place I’m spending that time, is this what I’m required to do? And is this for the necessity of the program? And is it good at help the long-term success of the program?

Rhea Frondozo: Right. Because I think goes to a second point to make about a potential pit fall that a breath TPM may have. It’s knowing when, when to rely on an SME versus is doing something yourself. Right? So it’s important to, for us as TPMS to understand a problem at hand, but knowing how deep do you need to go in that problem and how deep you need to go in the solution versus making sure that you’re pulling in the right people to do it, or just being the person that vets, are we solving it in the right way or solving the right problem.

Because at the end of the day, it’s not your job to be the SME, but it is your job to know when to utilize the SMEs for their experience and how to gut check them, to make sure that what they are delivering actually meets the business need.

Mario Gerard: Yeah. And sometimes these SMEs, like they over-engineer things, or they will complicate things, right. You’re laughing and I’m laughing. Cause we’ve been through this situation so many times. And we bring in an SME, who’s so focused in the area. They know it’s so well that they sometimes really, really over-engineer and overcomplicate things.

Rhea Frondozo: Not only that lot of times, what will happen is these goes to this, the thing that you mentioned earlier about scope creep, maybe there’s this problem that you recognize impacts to your program, and you need them to solve it. Oh, but this has been a pain point for them for the last, how many years. And now this is their chance to get buy-in to make this awesome solution that will fix everything for the problem that they have, which is only a portion of the problem that you have.

Mario Gerard: You need, you need them to solve X, but they need to solve X in hundred. That that’s their biggest customer problem, which they’re trying to solve, but they haven’t gotten an executive buying for that, but they wanted to stack it on.

Rhea Frondozo: Right. They want to, I’ve seen the, has happened multiple times where essentially, they it’s like they want to ride your coattails. Oh, you have a priority for working on this thing. Oh, that means I can get the priority for working on this whole entire thing. Because it’s associated with your program. Yeah. Which again, this ends up becoming scope creep. That ends up being a lot bigger than you really need.

Mario Gerard: Yeah. Yeah. So, it’s very interesting, like two common pitfalls. Why don’t you talk about like one of the programs, which you worked on Rhea and kind of like walk us through like the complexity, the, the objective and how you went through that.

Rhea Frondozo: Sure. Yeah. So, I’d say that the biggest and most complex program that I’ve ever owned was running the global government sector program for OCI. And this encompassed a suite of cloud regions, services, features, and processes that were needed to meet our government customer expectations.

Mario Gerard: So, it’s all like a new product.

Rhea Frondozo: It’s actually a set of products.

Mario Gerard: It’s a set of products that enable a particular government to run on our own infrastructure.

Rhea Frondozo: Right. And the reason why it’s a set of products is because you know, for governments, you’re not only looking at running on public cloud or public workloads, but they have different levels of classification. So, we had unclassified levels of workloads that they may want to run. Or you also had classified levels that are running at top secret or secret levels that require much more engineering to secure the workloads that they need to run.

Mario Gerard: And when you talk about engineering, just to elaborate a little bit, it more is physical infrastructure needs to be re-engineered people need to be re-engineered because they have certain level of clearance, the way they operate on the cloud differs the software running on the cloud differs. So literally every single thing has to be re-engineered meet this product need.

Rhea Frondozo: Right. And so, you know, the way that we had to think about it is what is the lowest common denominator, right? So how do we make sure that we have the highest level of security in a product and making sure that we run it everywhere that way so that we try to minimize that divergence.

But it did mean that we had to reengineer the products, not just for the government customers that we had or the government regions that we had, but all the way through the entire stack, down to the public versions of it, because we wanted to be able to minimize the divergence of software.

Mario Gerard: This is a very, very interesting point that we didn’t speak about this whole podcast. On all the pro we worked on, we try to intro divergence is limited. And especially when you’re working in an organization at the scale at which we worked in like several thousand people strong, right. Divergence is a very important thing that we try to avoid. Tell me, why is it so important? What is divergence for maybe for our listeners? Can you give an example of divergence? What do you mean by divergence?

Rhea Frondozo: So, you know, by divergence, I mean really the code that’s running in any… Or the way that you are operating these regions is important that we do it the same across all regions. Otherwise, the amount of work you are creating for yourself can double or triple, a quadruple, because now you have to update multiple code bases.

You have to have different processes running for different regions. You have to hire different people to do all the different work across all these different regions.

Mario Gerard: And this simply multiplies the complexity, every single team supposed, you have 200 different teams for 200 plus products. Every single team will need to operate in a different way, and you have one garment, then you other garment, then you have another garment. And so, your divergence level becomes increasingly astronomically high. And this is just not this particular problem Rhea, and I are talking about I’m, I’m talking about this.

Whenever you have any kind of large-scale program, you need to keep in mind that you’re not introducing too many divergent processes or tools or architect so that the teams don’t have to carry that burden forward in having this different way they act for this particular type of problem.

Rhea Frondozo: Cause what you’re doing is trying to make sure that the products that you are creating are scalable.

Mario Gerard: Yes, yes. This is what we always talk about. Is the process you’re creating scalable? Is the architecture that you’re creating is that ability to apply that all throughout the organization, otherwise you’re going to have just too many people.

Rhea Frondozo: What you don’t want to end up with is armies of people, all doing different things. When you could have architected your solutions in a way that allows you to scale to multiple regions without having to hire an army for every region.

Mario Gerard: Yeah. So, sorry I took you on that divergence question, but I thought that was a kind of a very important point, especially when you’re running large skill programs, but do carry on.

Rhea Frondozo: Okay. Yeah. So, the global government sector program is the largest program that I’ve owned. And when you think about the sponsors and the stakeholders that were required for this program at the end of the day, the customers were the various different governments, whether it’s foreign or domestic. And we had very specific sales organization that we were enabling by providing these products.

These were, you know, sales orgs that were the interface and to our government customers who are going to be the ones to manage the RFPs, the request for proposals and the responses that we would provide. And, you know, we’d work very tightly with them to understand and interpret requirements and provide our technical assessments of any gaps that might be missing or products, feature gaps that are, are required. And then we had the executive sponsors for this program and really this one went all the way up to our CEO at Oracle. In this case, it was Safra Catz. And this was a companywide initiative that affected all services that we delivered.

And ultimately me making this huge sales opportunity because we wanted to be able to sell all products to these governments.

Mario Gerard: And when we talk about all products, it’s kind of very interesting from an Oracle standpoint, right? Because Oracle has like more than 300 different products and government might use any of these 200 different products. Even if you cut it by half, they’d use a hundred products plus, right. And you have to ensure that these hundred plus products are going be able to run on an absolutely new type of environment.

They’re going to be able to deploy it, patch it, maintain it, or operationally manage it in this new type of environment.

Rhea Frondozo: Right. And the thing to mention there is that was of the utmost importance cause the customers had experienced where in one of our competitors, they had divergent code bases, which made it very difficult for them to scale in a way that allowed all products to enter their government re regions. And that’s exactly what our government customers did not want.

So, this is why we have to reemphasize the point of the importance of why we didn’t want to have divergent infrastructure and technologies running in these regions. So outside of the, you know, executive sponsors, we had really the entire org who are responsible for not only building the regions, but delivering the services and the features in these regions or creating new features that were needed to ensure that we met the government security requirements and that we were able to maintain parity with every other public cloud offering that we had.

Then last I’d mentioned in terms of key stakeholders were operations. You know, we had to have specialized operations teams that had to engage with this product in a way that met security standards as defined by the government. So, you know, it meant that we had to have appropriate security levels, clearance levels and operate following, you know, very specific data sovereignty rules and how we manage and access customer information.

So, these were some of the core stakeholders that we had to work with. And when it came down to running the program and running these programs and these communication mechanisms to do so there was a lot of planning involved when it came to creating the internal kind of vision for what the product was going to be defining, you know, the product definition, defining what the resourcing requirements were to do this in this case As I mentioned at Oracle, we are very big on doing write ups as opposed to PowerPoints.

And so, we started with very detailed product definition, you know, six pagers. And once we were able to get leadership buy-in then we were able to summarize these requirements and these goals for the program in what we ended up doing as a power or point roadshow where we then went to all the different leaders in the org explained to them what we were doing, making sure that we had buy-in at the next leadership level. And so that we could ultimately get the pocs assigned to our program.

And then we held bigger tech talks for the entire engineering organization, just to give everybody a heads up of what’s coming the importance of it. So that the next time we came knocking on your door, making an ask for a particular feature or a particular new service, you knew what the importance was and why it was important that you prioritize it.

Other things that we ended up doing for this program were creating weekly internal stakeholder meetings, executive reports that we would share at, you know, critical projects, stakeholder meetings created newsletters and Wiki pages and confluence pages, the track, the project, the schedules, meetings, agendas, key requirements. I mean, you name it. All of the different tools that I mentioned earlier were all employed in this particularly large program.

And the thing was when I started the program, it started as building a net new region and the program grew to owning the entire suite of cloud regions that were needed for these customers. And so, it ended up being, I started owning the program in one phase, which was the planning phase of a new region. And then I ended up taking on more pieces of this government sector in terms of building other cloud regions for other governments like international governments for maybe the UK and these ones were in different phases, not just in the planning phase, but maybe these ones were already being built.

I ended up taking on other regions that were in different phases, such as they had already been built, but they were in the support phase now where the product had been built for the u’s government. But as customers were using it, now you end up in a situation where we’re finding what gaps are missing, what features they need, what new services they need and figuring out how to prioritize, getting those requests into these regions.

Additionally, these regions that we had built ended up taking the place of older versions of the cloud regions that we had. And so, we ended up in having another program that was essentially to close out the existing regions as another kind of program in a different phase. And so ultimately this program that I owned ended up being extremely complex because not only of the different number of stakeholders that we had to work with, the amount of business opportunity that it brought opening up multi-billion dollar bid responses. But also, the complex nature of owning multiple regions, products, and various different life cycles of a program phase.

Mario Gerard: That’s so complex hearing you speak about it. It’s so incredibly complex and so many different requirements, so many different teams to deal with. And I hope our listeners got that. This is kind of the end of this particular podcast. I’d really like to thank Rhea for the amazing, amazing work of giving us all this knowledge and all the knowledge she’s built, sharing that with us.

I’ve seen her in action and then she’s like unbelievably in running these kinds of large programs. And that’s something which we’ve done together and it’s incredibly fulfilling. There’s definitely a lot of challenges, but at the same time, it’s very rewarding as well. So, I hope. I hope all of you enjoyed that. Definitely share the podcast with your friends and colleagues and subscribe to the podcast on your favorite podcasting app.

Thank you, Rhea So much for sharing all that. You want to say anything, any words, anything that you want to add?

Rhea Frondozo: No, I think, you know, I’d just like to thank you Mario, for giving me the opportunity to share all of my experience and knowledge. I think this is being a TPM and gaining this kind of knowledge is not something that you can read in a textbook. It’s not something that you can study for and achieve.

It’s something that you learn through experience. And so, what I found is the experience that I’ve gained only comes through trial and error. And being able to talk about it now with you, hopefully gives our listeners the opportunity to kind of ahead of the curve in terms of understanding what to expect when they get into the situation themselves.

Mario Gerard: Yeah. Yeah. This is an incredible rollercoaster of a journey, right? So, thank you so much. I hope all of you really enjoyed that.

And that my friends are the end of three-part series on how to run large scale programs. I really hope you enjoyed that. This is one of those most unique podcasts, I think because there’s so much information, knowledge from somebody who’s so experienced in this field. I really hope you enjoy that. If you like it definitely share it with your friends and leave us a review on your favorite podcasting apps.

Thank you so much for listening. I’m going to be producing lot more podcasts. So, if you would like me to contact somebody who you think would be great for a podcast, let know, thank you so much my friends.