Navigating Challenges in Program Management | 10min snip from The Technical Program Management Podcast & Interviews

Speaker 2
You say it as if it was easy, but you have some of the best people in the world. It's like an excellent team that you assembled there. And what does it take to write Rust at that level? Do you have any coding guidelines internally that you stuck with? Is there anything internal where you say, other companies could also profit from that knowledge too things that we might avoid even for example overusing generics or mixing sync and async what are some of the patterns that have evolved when using rust at that level yeah
Speaker 1
i mean i also want to like i love my team. Everyone who works Sox at is great. But also like we are, while we are very senior engineers, almost universally, you know, there's not something super magic that necessarily makes us hands and shoulders better than everyone else. I also want to like, you know, I appreciate the kind words and I don't want to say my coworkers are bad at their jobs because they are not. They're definitely very accomplished. But like also, you know, it's engineering. It's not magic. So there are also many, many good Rust engineers in other places as well. What I will say is that when I started four and a half years ago, there was a lot less experienced Rust engineers in general, period, just in the industry. And so a lot of the folks that we hired at the beginning were not necessarily fantastic Rust programmers. They were experienced and fantastic engineers in general, but they sort of not necessarily picked up Rust on the job entirely, but let's just say we didn't have the luxury of demanding that people were good at Rust before joining Oxide in the earlier days. And so part of me coming on relatively early on was kind of replicating the you know you know if you commented on rust and hacker news in the last 10 years you probably got an answer from me personally at some point because just like i spent my time doing that and so in the early days of me working at oxide i would spend a lot more time answering rust questions helping people with rust stuff, giving advice and choosing packages. But we also did have a lot of people who had a lot of deep rust experience when I started as well. But like part of me coming on initially was to sort of help make sure that if people had rust issues that like I was able to help, you know, sort of guide them with stuff like that. By now, you know, years later, there are many more experienced Rust engineers out there. And so I would say that one of the factors that sort of happened is the overall quality of the Rust hiring pool has gone up. And so at this point, we are much more likely to hire someone that knows Rust than not know Rust, simply because there are enough people that know Rust at a high enough level that we're able to find and hire those people. So that's changed over time to some degree. But also, some things that we do that sort of any company that's doing Rust can kind of like replicate to help out with these sort of things. We have a dedicated channel in our chat system that's just purely for Rust questions. And so people will ask Rust specific questions. And I and many others who have a lot of expertise in Rust will make sure to pay attention to that question and answer questions as they come up. I think that's really important. Having space for people to ask things is really a big deal. We have a biweekly meeting that's optional. Other than one-on all employees can go to all meetings. So I want to say it's available to everyone, but that's just kind of true of stuff at Oxide in general. But if your company doesn't do that, I would still recommend a sort of like open to everyone sort of rust, we call it the rust study group. And basically, it's a meeting where, you know, every week, me and a couple other people who, you know, love rust stuff, have always blocked off on our calendar to make space for if somebody has a question that's maybe not hyper urgent, you know, or, you know, sometimes people do like, oh, I was working on this hobby project. And I came across, you know, I didn't want to bring it up at work in work hours because it's not really about work. But I do, you know, like, I've been learning this thing about Rust and I don't really know this detail or whatever, you know, that maybe helps them, you know, with their job and some other, you know, maybe it's not on a project that's job related, but like leveling up at Rust is still going to be good, you know, for your job. So people will come and bring questions about like, you know, Oh, Hey, I saw this thing go by earlier in the week where people were talking about this new feature. And I was curious what people thought or how it works. Like, I don't really understand why people care about this. Or, you know, maybe they're like, I have this tricky bit of code that I can't really figure out why the bar checkers mad at me or like, you know, whatever. And so, you know, we kind of have both of those forms of sort of explicit time and space to help people with Rust related problems that they kind of have. In terms of specific things that have popped up, I think the biggest thing that's kind of, we've talked about it a little bit, but we haven't really done as great of a job of like, getting our perspective on the conversation out there. But a really big thing that's popped up with us that some people are talking about for sure is async cancellation and so for those of you who aren't familiar async rust has what i think is a really great cancellation model conceptually which is like futures don't do anything until they're pulled and to cancel a future all you got to do is just drop it and then it won't ever be pulled again and so it's effectively canceled and that's like cool but that also means that there are subtle issues where you don't realize that a future can be canceled because like dropping something is not obvious in rust all the time and so there are some patterns you can get into where something will cancel and you don't realize it and that will lead to sort of like logic bugs and so this is sort of an area where we've kind of like we we it it's a thing that surprises people because you know if it compiles it works is not literally true but it definitely can feel like it's true sometimes and so a lot of people's experience with rust early on is like oh my god it catches all these problems at compile time and i don't have these bugs because i used to have to always worry about you know like them happening and now the compiler catches them for me and that's great and they go into async and they hit their first time where something gets canceled where they don't expect it to and it feels it feels like walking that back they're like back in the land of like oh my god the compiler doesn't catch this problem for me anymore and now I feel like, you know, these like promises of Rust catching all these things are like a little not as true as they used to be. And I think some of that is just like a natural counter reaction to how much Rust handles for you in most cases. But it is true that like async cancellation issues are like not generally statically findoutable. And they also can be surprising. And that becomes itself surprising when you're sort of so used to Rust catching things aggressively. And so we've had to try to find some places where we can figure out like, okay, here are some patterns to be avoided or here are some libraries of like trying to be a little more explicit about cancellation. I'm totally drawing a blank off the top of my head, but there's been some experiments that some of my coworkers have tried to do to sort of like figure out some patterns like that. And we've wanted to kind of like talk a little more about these problems publicly, but have not actually had the time because it's always super busy when you're doing a zillion things to like necessarily get out there with all their experiences. But definitely it's a thing that the async folks are familiar with being an issue overall. And I think that anyone that does Rust async at scale has eventually either found out about these bugs or maybe you've had some mystery bugs you can't track down that eventually you'll discover cancellation related. But that's definitely an area where we've found stuff to be a little surprising and have started to get out there a little bit and talk about sometimes.
Speaker 2
You said the drops are sometimes not obvious in rust and that can lead to subtle bugs with phasing execution can you give me an example for that yeah
Speaker 1
so i'm gonna be very vague because to be honest this is not an area of the product i work on personally and when i was in these conversations was more like six months ago so i'm like a little teeny bit out of the loop but let's just say like a big thing is like, a concept in this area is like if something is cancel safe. And what that means is sometimes you have some code that is like, if you drop it in the middle of its operation, you need some sort of cleanup or you need to do some sort of thing or notify someone or something. And so that won't necessarily happen if an operation is canceled in the middle of it occurring so for example like say that you're doing a select between some sort of like receiving on a channel and then you're also like printing you know you're sort of selecting between some sort of timeout and some sort of receive on the channel like that if if like the if the if the timeout finishes before, you know, you're calculating some sort of value while waiting on the like receiver, then like that means the other future in that select gets dropped. Like, you know, like if you're doing a select, it kind of inherently means that you're going to drop the other things that did not finish early. And so depending on how you've written those futures, like obviously waiting on a channel and you drop it, that's fine. You're just no longer listening on the channel. And say you're just sleeping and you drop that, it's fine. You're no longer sleeping. But say that the thing that happens first is the timeout. Maybe that calculation had some sort of thing where it needed some sort of cleanup step. And now because it's timed out, that future just gets dropped. Well, if you didn't do something to ensure that the cleanup would happen properly, then maybe that doesn't necessarily work. And so this is like code that is totally fine. If all the futures are like cancel safe, if they have some sort of ability to do, you know, that kind of thing. But you know, if they don't, then that can kind of like, be an issue. And so, you know, that's like, yeah, like, you can lose data sometimes. So for example, say, you know, in one future, you're kind of like writing some data inside of a loop. And then that that gets canceled in the middle of it. Well, that maybe means you only wrote part of the data, but you never wrote the final part of the data. And so that's like an example of like a problem that can occur if something is not necessarily like fully cancel safe. And so, you know, this is where a lot of the discussion about async drop comes in. And like a lot of that stuff, if you've seen that discussion and sort of the Rust async world. And so, yeah, I don't know,
Speaker 2
I guess that's like a high level example of what's going on. Can you quickly explain what async drop means?
Speaker 1
Yeah, so there's a drop trade in Rust. And when something goes out of scope for the final time, then the drop trade gets called and that runs some code. This is like into a destructor in many other languages, like especially OOP languages and definitely conceptually sort of similar, but in the async world, if a future doesn't get pulled anymore, there's no equivalent like hook, right? Like the drop trait kind of gives you the ability to sort of hook into, Hey, this object is about to be destructed or destroyed. So like do something then. So like a classic example is box, you know, you allocate memory up front. And then when drop runs, you know, it would deallocate the memory. So a future doesn't have any sort of similar kind of mechanism. Like if you would allocate some memory manually in a future, and then it no longer gets pulled, now that memory is leaked, because there's no equivalent hook and point to sort of say, destroy that memory that was called. And so there's been a lot of different proposals for what an async drop trait could look like, and how that might happen. And, you know, or how we might want to solve the problem in a different way. You know, there's been sort of a discussion in the async working group for a couple years now. Why
Speaker 2
can't you just implement drop for the future because
Speaker 1
drop is a synchronous it is a function not an async function and therefore you're sort of like calling something synchronous inside of the you know and plus the other reason is like the definition of dropping it like it's no longer being called anymore like kind of is a fuzzy definition. Like if I don't call, if I don't call poll for an hour and then I call it again later, you know, like technically I've called it again, but during that hour, I may never know if I'm ever going to be getting it called again. You know what I mean? And so there's also a little bit of like fuzziness there in the definition. And it's much more straightforward with synchronous code, because there's no, like, I mean mean you might have a sleep call or something but just like the point is you can always statically know like okay eventually this point is going to happen where it's never going to be used again and in async stuff that can be a little more tricky and or like dynamic but you know also this is an area i work on specifically so i can only really give a higher higher level answer on that yeah i
Speaker 2
think the slippery slope here is that when you go forward with that proposal or you go forward with that idea of introducing async equivalents of sync traits then you end up maybe replicating parts of the standard library we see some something similar with read and async read or write and async write yeah
Speaker 1
i mean the thing is is that programmers love like dry and they love trying to unify concepts that are that seem similar but are different and i think that's true but i also think that sometimes similar operations are just fundamentally different and you can't really abstract over them in the same way. Async read and async write are great examples where we have the sync write and sync read APIs. But the thing is, is that like for synchronous read and write, those APIs are very straightforward and have existed for a very long time and they like make sense. And there's only one implementation of them. one of the reasons why async read and async write have not been stabilized yet in rust is because there are at least two meaningfully different proposals on how to implement those apis and so yeah well conceptually it's like just read but async i think that but is really load bearing and like you know like this is maybe stretching an analogy a little bit too far, but like addition and subtraction are both conceptually the same thing at some high level. Like they're both a binary operation with an operation in the middle. And you want to be able to say like, well, you know, like maybe those are the same thing. And so we should abstract over both of them. And that's how you get monoids. But like, not monads, but monoids. like that doesn't always mean that abstraction is useful necessarily because sometimes it can paper over the details that matter you know like at a high level the idea that like i'm applying a binary operation to two things and it does something is like sure sometimes working at that level of abstraction makes sense but sometimes you really care if it's actually addition or actually subtraction you know I mean? And so I think that a lot of people want to reach for the idea that we should inherently be able to abstract over sync and async. But I think that there are different enough things with different enough semantics that doing so at least, let's put it this way, for Rust specifically, a language that cares about the low-level implications of what you're doing, that you need to be able to integrate with an underlying system, that the details matter to you. I think that over-abstracting these things is a mistake. I think that in a language like Haskell, as a reason arranged for Monoids a second ago, in Haskell, you extract over-sync and async because conceptually they're the same thing but haskell does not have the same performance requirements and like low level requirements and little commitments that rust does and so it can it can afford that abstraction because that abstraction costs you something but in rust i think that the the details are significant enough and the processes are significantly different enough that it is important and meaningful to keep them separate and so i don't think it's inherently a bad thing that you're sort of redoing some async stuff or some sync stuff that's in the standard library because like also drop conceptually makes sense i'm not sure if i fully think that async drop specifically is a good idea but some sort of analog or way to solve that problem makes sense i think that read and write are there true but like that's like 95% of it. There's like not a whole lot more, you know? Like we're not necessarily like, the other stuff that we may like make async is like still like, I don't know. It's not a brand new concept or like, I don't think it's the, I don't think it's the end of the world. Let's put it that way. Some duplication is totally fine and meaningful sometimes you know abstractions are good because they let you work with stuff at a high level but you also need to be able to do stuff at a low level and i just like think that sometimes that trade-off is that you don't get to use the high level abstractions so but this is definitely like a really big debate in the rust world right now but
Speaker 2
at the same time couldn't agree more you really summarized it super well because this is one thing that i really love about rust it's explicitness and not having these leaky abstractions because you make it explicit that there is a difference between those abstractions and and i really like that part but at the same time the async is pretty new. The rest of Rust, the sync part, has matured in the last 10 years. And since I have you, I might as well just ask you, because you have a lot of experience, you've been in this community for a long time. What would you say is one big mistake that the Rust language made? Something in the standard library or anything regarding the syntax or its semantics that you would probably see as a historical mistake and you would want to change yeah
Speaker 1
so i have a joke answer that's funny and then i have a time where i thought that happened but i was wrong and then i definitely have a good answer for one that's real but i'm gonna tell the first two anyway to like let me think about what it actually is because like you know i want to make sure i'm getting a good answer so the joke answer i always say is that string should have been named stir buff and like that's and that's just like because like we have path and path buff like i think that naming and the fact that it's like capitalist string like i feel like there'd be a lot less like rust has 36 different string types if we like acknowledged a little more that like it's a buffer, like it's a mutable growable string as opposed to some other kind of string or whatever. And so that's what I've always joked is like I want Rust 2.0. I don't want anything changed except for I definitely like want stir buff instead of string. And so that's kind of like a silly, silly answer. There was a time where I thought Rust was making a huge mistake and i was definitely wrong and that is the post fic post fix await syntax so that was like for people who weren't around when we were doing async awaits design like there was a point in time where it was a hugely contentious topic about how to write await like should it be a prefix thing like javascript should it you know be some sort of other thing or like should it be what it is today which is like the dot await and i am really conservative when it comes to programming language design actually and i kind of like at the time i was like we have a lot of people are coming from javascript they and javascript and c sharp both do this prefix await shenanigans and i think that like when it's not clear which way you should go you should choose to be conservative when it comes to language design in many other way things in life i do not believe this but like when it comes to programming language stuff i think being conservative is generally pretty good and so i thought it was a really big mistake to add this weird syntax for async await. However, after having had to write a whole ton of async code, that would have been a huge mistake. And I'm really glad they did not listen to me personally and went with the postfix await anyway, because it's just clearly superior in every possible way. And brilliantly enough, people figured out how to address that major concern that I had, which is like what happens when JavaScript people come and write the wrong thing and that is diagnostics in the compiler so a really cool thing that rust does that doesn't have to do is sometimes the rust compiler will parse code that's wrong to give you a great error message and so if you write you know instead of foo.bar if you write await foo.bar, the Rust compiler knows how to parse that, even though it's not write Rust, just so it can deliberately say, hey, this is not how you write await, you would write it like this, foo.bar do that instead. And that is like a really trivial way that like anyone who is like writing their old style of, you know of coming from another language they won't get confused they will get helped into writing the correct thing and so that's an area where i definitely was like at the time i was like on the wrong side and i think i've like thoroughly admitted that that's like a mistake i think my biggest things and started Rust made mistakes that I think is a little more serious or real is there was a couple things that sort of like, I don't want to say that no one cared about them at 1.0, but there's some things that were sort of designed in a certain way that there was so much work to do that they never really got fully completely totally thought out and i think the biggest one of those is the module system i i really like rust module system i think a lot of stuff that it does is good but it is a common problem for people coming to rust and it has been and in rust's 2018 we did some changes to some things that made it a little bit easier to understand but like it is the number one thing that when people read the book and they say i was confused by something they say the module system and i don't know what that is i don't have a constructive necessarily answer for how we should fix things instead. But like, it wasn't, it was something that was kind of built a very, very long time ago. I don't even remember by who initially. And then like, there was just so much other stuff to do leading up to Rust 1.0. There was never a moment where it was like, okay, we need to really make sure to think through, is this how we want the module system to work? And then even in 2018, when we did, there was a bunch of like legacy constraints of like, okay, we want to change how some of this stuff works. But we have to make sure that it's not too different for all the people that currently know Rust, because that would be like very, very bad if there was two completely, totally different systems. And so some of that was kind of like tied up that way. And I think in general, it's not really just the module system, but also like name resolution in general, which is like when you type an identifier, how does Rust determine what identifier you mean? The module system is like part of that whole situation. And a lot of that is kind of just like was made and was never really, it never really had the time to be like fully thought through and designed by someone before 1.0 came out and i think there's a way that you could simplify a lot of that stuff like this gets into stuff that like most rust programmers never even really think about but like types and modules are two separate like name spaces and so you could have a module with the same name as a type and the Rust language will disambiguate between the two and it's totally fine. And there's like, I think there's another one too that I'm totally drawing a blank on that does like three different sort of like versions of namespaces and they can all be like overloaded. And this like leads to like very confusing outcomes if people name things in sort of strange ways. And like that's all complexity that the compiler has to deal with when trying to look up and figure out a name is like looking through all those things. And I think all of that probably could have been pretty radically simplified and is something that's like basically kind of impossible to sort of like deal with at this point. Like it's just kind of like baked in. Macros also are kind of like nick cameron really wanted to do like a rebuild of macros and so like that's why before russ 1-0 we renamed the macro keyword and macro rules and reserved the macro keyword because the idea would be that like someday there would be like a new macro system that would be kind of like built and that's just never happened like that all of like disappeared at some point and so you know that's definitely like a thing that's kind of similar although i don't think most macros are inherently sort of bad but like there definitely are other options that could have been considered a little more and like proc macros are like sort of wonderful but they are also really complicated and have some like really weird technical angles like you have to make a proc macro its own crate why do you have to do that well because proc macros are kind of like an out growing of compiler extensions which existed a long time ago and like why were compiler extensions the way they were well back in the day it was basically like you can ask the compiler to just like open up a library you write and mess with a compiler internal data structures and produce something and like proc macros are a crazy strong aspect of the rust ecosystem but that doesn't mean the feature had to be implemented the way that it's implemented and now that's like kind of just there forever and also it's good enough that there's nobody who's like truly invested in making a better one. And so, you know, like that whole area, yeah, again, is like sort of a similar, like, you know, is it the worst? No. Could it have been better? Yes. And I think those are kind of some things that like a Rust++ could like meaningfully address. I don't think they're necessarily enough to justify having a whole separate new language, but they're definitely areas that I think that are big warts that you know maybe could have like been fixed but just like aren't really possible to do now all
Speaker 2
right

TPM Podcast with Rhea – Episode II Part I

The Technical Program Management Podcast & Interviews

Remember Everything You Learn from Podcasts