Navigating Growth and Challenges in Solana Labs

Anatoly Yakovenko (Solana Labs Co-Founder) joins The Zeitgeist to discuss the current state of Solana and the ways in which the team is working to improve protocol performance.  

SHOW NOTES

00:39 - Origin Story

01:59 - How his role evolved at Solana

03:42 - New initiatives at Solana

04:59 - NFTs

06:37 - How to improve performance

12:31 - What is Solana's North Star?

15:48 - Growth and Nakamoto coefficient

19:47 - Developer ecosystem

23:31 - Craziest idea built on Solana

24:35 -  Builders he admires

REFERENCES

Saber

Mango Markets

Orca

Anchor

FULL TRANSCRIPT

Brian Friel (00:00):

Hey everyone, and welcome to Zeitgeist, the show where we interview the founders, developers and designers who are pushing the Web 3.0 Space forward. I'm Brian Friel, Developer Relations at Phantom, and I'm super excited to have on the show today, none other than the man, the myth, the legend himself, Anatoly Yakovenko. Anatoly, how's it going?

Anatoly Yakovenko (00:22):

Hey, it's going well, thank you for having me.

Brian Friel (00:25):

Thanks for being on the show. You know, I think most folks know your story if they're very familiar with the Solana space. You are the founder of Solana. We do have some people who are listening to this show who are more familiar with the Ethereum world. Would you mind giving a quick moment to walk us through your background and how you got started with Solana?

Anatoly Yakovenko (00:40):

Yeah. So I'm an engineer by trade. I spent most of my career at Qualcomm, and I was there from super early days working on these like flip phones, if anybody remembers those. I was a Core Kernel engineer on Brew, which was, I think one, one of the first mobile operating systems with an App Store. Really the first, I think mobile platform where developers build anything, and this was pre-iOS, pre-Android.

Anatoly Yakovenko (01:05):

I was there for like almost 14 years. My last project was like building optimizations for augmented reality. And then around 2017, this big boom and crypto happened. I think the first really big Bitcoin price movements happened during that year. And it was obvious that Bitcoin was too slow for the kind of stuff that the narratives that people were talking about at the time, such as like global payments and things like that. That's what really got me interested in the space because I saw an opportunity to apply all the things that I learned at Qualcomm, from optimizations and wireless protocols and everything else to crypto.

Brian Friel (01:46):

So in the early days, if you take a look at the Git commit history on Solana, the protocol, you were a pretty heavy code contributor. How has your role evolved since starting the project? And what would you say is like your main focus today?

Anatoly Yakovenko (02:00):

Yeah, those early days were hard because I had to raise money and kind of be the CEO, which is effectively sales, but also I'm a really strong IC and losing one IC, it sucks, right for like a small startup? So I was coding and sometimes like 2:00, 3:00 in the morning, I would just be awake, still building stuff and burning the candle on both ends.

Anatoly Yakovenko (02:27):

As the network got stood up around like 2020, once we started growing the team, when we got a bit of traction in 2020, it's just my initial contributions were not as impactful as ... I'm now like one out of 20 contributors instead of like one out of 10. And that amount of like effort that I can put in ended up being not as important as me doing everything else that I can do as like the CEO of Labs.

Brian Friel (02:53):

So you guys obviously had a very explosive summer, 2021. NFT trading on Solana, I'm not sure if you would've predicted that your global state machine would be the perfect mechanism for trading JPEGs on the internet.

Anatoly Yakovenko (03:06):

No. Yeah, that's been wild.

Brian Friel (03:10):

I would say it's funny because a lot of people initially wrote off Solana because it wasn't EDM compatible. There was a myriad of reasons why it wouldn't take off. It seems like today you guys have the opposite problem where there's so much demand, everyone wants to use it.

Brian Friel (03:25):

I know you have a couple initiatives in the works now, some of that being quick, some stake weight, and then there's a fee prioritization mechanism coming in. How are those initiatives going generally? And do you foresee this as being a big step for the network or is this a really an iterative phase where you guys are going to continue to scale in various projects?

Anatoly Yakovenko (03:43):

Yeah, we designed this network. I mean, initially the slide deck literally said blockchain and NASDAQ speed and there was this big vision of something like serum running on it and being the price discovery for all of the world's stuff that needs trading. And that was like the first use case that we actually had any success with.

Anatoly Yakovenko (04:01):

The folks from FTX, that incubated serum, they looked at the network and they were like, "Okay, this is designed for the kind of exchange that we know how to build and trade. So why don't we see what happens?" And when NFT started taking off, it didn't make sense to me. But I could tell that a lot of people, like humans were interested in them instead of trading. A large number of people were interested in them. And that's a really important thing, like product market fit can be measured in many different ways. If you're speaking trading it could be volumes or TVL, but it's really, really important to really connect it to a person.

Anatoly Yakovenko (04:39):

And the more people you have in any of these effectively web based networks, people want to call it Web 3.0 or Web 2.0, or whatever, it's still the web. The more people you have using it, I think the more value you're actually creating for the world and that's really, really important. So Metaplex was incubated in Solana and it just allowed people to launch NFT projects.

Anatoly Yakovenko (04:59):

And we saw that at that time and now Ethereum gas fees were really so expensive that it was effectively impossible for any small artist to pony up like hundreds of thousands of dollars to go deploy like 10,000 NFT project and mint them and like have their users pay these ridiculous gas fees.

Anatoly Yakovenko (05:20):

So a lot of people just started messing around on Solana and they started getting traction and like making more money than normal artists creators make in a year in a single NFT mint. And that really changed like everything, for them and for us. So that's been just awesome to watch. Because NFTs have value, that has led to a different kind of traffic than we anticipated where there's now bots that spend a lot of money on egress, like a hundred gigabit worth of egress to try to snag a NFT mint.

Anatoly Yakovenko (05:55):

And the network can handle that most of the time. But sometimes it can't. So now we've been working on – if you've been following Solana, I'm sure you've heard like there's outages or congestion. Basically what happens is you have a known event, financial event, like an NFT mint or an IDO, but most of the time it's an NFT mint. And bots start sending so much traffic before this event to try to basically maximize their probability of them getting all the NFTs or most of them by stuffing all the leaders with packets and that prevents other packets from landing.

Anatoly Yakovenko (06:31):

And sometimes like the last one, it may uncover some unknown memory issue, where the memory and the validator starts growing and they start shutting down. And therefore, when more than a third of them run out of memory, you end up losing a quorum and therefore consensus halts. So it's effectively the greatest bug bounty free, like DDOS testing, not the best kind of DDOS testing that money can buy.

Anatoly Yakovenko (06:58):

It's really, really hard to even get a hundred gigabit worth of traffic like at Google. Your typical network that you deploy at Google for your private network inside their servers does not go to a hundred gigabit, it goes to one gigabit. So the kind of things that you see in the wild is really, really, even hard to simulate.

Brian Friel (07:15):

Yeah. You're testing and prod essentially.

Anatoly Yakovenko (07:17):

Right. And that's always the case. We have like a big testing effort and team and like a massive test net that's even bigger than the main net to try to simulate all these corner cases. Everything that you do in the lab gets really tested for real when you ship to prod.

Anatoly Yakovenko (07:37):

So the major change that we need to do is actually prevent them from sending that many packets. And there's a bunch of ways to do that. TCP has been the classic solution, but it has a lot of problems with latency and managing connections.

Brian Friel (07:50):

Right. And you guys use UDP Today.

Anatoly Yakovenko (07:52):

We use UDP Today, which is what traders prefer. So all the exchanges, like when you get high connectivity from CME or an ISE, you get UDP. And you can manage it for most of the time, but at some points like at a hundred gigabit, you basically have to like decide, "Okay, we're going to spend 10,000 bucks, a validator and put these appliances in there that can drop maybe close to like a terabit worth of traffic really, really quickly and filter packets. Or we build a software solution."

Anatoly Yakovenko (08:24):

In theory, QUIC can be as fast as UDP. In practice, that implementation and testing it, rolling it out, takes time. So over the last release, the 1.10 release, our engineers, plus the folks from like Blockdaemon [inaudible 00:08:40] one have been working on rolling out the quick implementation. And it looks like finally, like 1.10 is going to roll out to main net.

Anatoly Yakovenko (08:47):

I think the slow release is going to, roll out is going to start this week, but you need like, you really need three pieces. So this is kind of a complicated problem. So if you have QUIC, you can limit how many packets any IP address can send, but bots can, then we were going to get a million IPs and they'll all send a couple transactions per second each because they really want those NFTs.

Brian Friel (09:08):

Yeah. And then you're playing Whack Em' All with the IP addresses.

Anatoly Yakovenko (09:11):

Right. So the second fix, that's also 1.10, but I would say it's not as perfect as I would like it, but mostly there, is being able to limit the amount of traffic; people send an aggregate by stake weight. So if you're not staked, you get the least amount of bandwidth. But if you are staked, then you have let's say 0.5% of the stake, you're guaranteed that all the other stake are on stake notes, can't star view. So you can at least send half of a percent of bandwidth at any time. So that's the second part.

Anatoly Yakovenko (09:45):

And then the third part is that as information moves from, let's say, RPC nodes or other nodes, and like propagates to the leader, including like these intermediate hops, we need to make sure that it's prioritized not by first and first out, because then again, that will also lead to bots just trying to be the first packet in any given event, but by fee.

Anatoly Yakovenko (10:10):

Right? So this is where we need to add fee markets, which is something that will force bots to spend money instead of packets. And it's a bit more complicated to add fee markets in Solana. If you've ever programmed in Solana, you'll quickly realize that it's really different from Ethereum. And the main difference is that – you have one giant state machine like you do in Ethereum, so there's no charting on state, but each transaction has to specify what state it's going to read and what state it's going to write. Which means that you don't have this global fee for a block. You technically kind of end up having fees for state.

Anatoly Yakovenko (10:45):

So you have a single NFT mint with a single account, like a state account that represents that NFT market. And everybody wants to trade on it. So the fees accessing that particular state need to go up because you can only schedule so many transactions to access that thing. But that doesn't mean that the block is full. You can actually start adding more events into the block that don't touch that part of the state.

Brian Friel (11:11):

And you can touch other parts of the state in parallel and at a lower fee.

Anatoly Yakovenko (11:15):

Exactly. So it's kind of like, think of it as a bunch of buckets, and you want to fill the highest contented bucket with the highest paying transactions, but then you also want to fill the other buckets. So it's this multi-headed queue problem. There's no perfect solution for it. So it's, every solution is a heuristic. Which makes it both kind of complicated and easy, that you're never going to be perfect. So you can ship stuff. You can't ship things, but because it's heuristic, there's always like corner cases and you got to be careful about if any of them can be exploited between all of these hops.

Anatoly Yakovenko (11:52):

Partial solution to that is also rolling out. But I think given how complex all these three pieces are, we need like, we'll probably need the 1.11 release to stabilize everything.

Brian Friel (12:02):

That makes sense.

Anatoly Yakovenko (12:03):

I'm glad that at least like these 0.XX versions of this stuff is out and like finally is actually rolling out.

Brian Friel (12:09):

This kind of highlights what I always found Solana to be refreshing, especially compared to older blockchains. You know, you mentioned Bitcoin when you were first in back in the day is that, you guys have taken this very practical approach to “how do we scale?”, and have these iterative solutions. Do you have like a defining lighthouse that kind of guides this development cycle? Is it about being like the fastest and the cheapest chain or what is your priorities here?

Anatoly Yakovenko (12:32):

So there's this fundamental idea like that goes back to the roots, blockchain and NASDAQ speed. Imagine like all of these computers around the world, they all hold state. And that state is synchronized within a certain amount of time. And that time in theory can be as low as halfway around the world, like about 120 milliseconds.

Anatoly Yakovenko (12:52):

So this is the one giant piece of state that represents every price in the world. And it's a point of reference for price discovery. And in theory, even if you have something like NASDAQ or CME that trade at like sub one microsecond, because news still has to travel around the world. You have some event that happens in Singapore. It's got to go through the same fiber cables as everything else. That news wire is going to travel at the same speed as a state, as a transaction in Solana.

Anatoly Yakovenko (13:21):

So by the time you see it on your Bloomberg terminal and CME, you'll actually see the price reflected in both markets. And that means that like this set of nodes run by volunteers with open source software can be competitive at the most important core part of finance with like NASDAQ with CME.

Anatoly Yakovenko (13:41):

So that's really the goal. Can we get it to that level where it's propagating information around the world, synchronizing it at the speed of light? So that's the north star. So ...

Brian Friel (13:51):

And I love that. That's great.

Anatoly Yakovenko (13:53):

It's hard. It's a hard problem, but it's, I think there's clear reason that this thing is really useful to the world if it exists and even in the state, it exists now, it's extremely useful because I think this idea of open cheap price discovery is really important for DeFi and Web 3.0. You look at something like NFTs. I think under the hood NFTs are DeFi, you have a smart contract that decides who gets what based on royalty, secondary trading and all these other things.

Anatoly Yakovenko (14:24):

And it's a different business model for making money on the web. It's one that doesn't involve the Ad Exchange, a centralized Ad Exchange that Google runs. It doesn't have any intermediaries. It's between the creators and the community that they're serving. So I think it's, as early as it is, and I'm looking at these things and thinking these are like early bulletin boards, I can see that I think in 10, 15 years, the way that people make money on the web is going to be 99%, no intermediaries like through these kinds of networks and applications and almost none on through like the advertisement channels.

Brian Friel (15:02):

Yeah. That makes sense.

Anatoly Yakovenko (15:03):

That to me, I think is like huge. If this like transforms the web, that's awesome.

Brian Friel (15:09):

Yeah, totally. Changes human behavior, fundamentally. So you spoke a lot about the human side of these people who are using the network, but there is a human side also to the people who are running the nodes that secure this network here. And there's a common misconception out there that Solana is this big and fast chain, but there's a few nodes that's powering this network.

Brian Friel (15:30):

You've spoken a lot about this term, the Nakamoto Coefficient, which is essentially the number of nodes that would have to collude to make the blockchain stop working in some capacity. How has this Nakamoto Coefficient been evolving since Solana's launch? And is there anything that you guys are doing at labs that are catalyzing this kind of growth in this Nakamoto Coefficient?

Anatoly Yakovenko (15:49):

Yeah. We started with like the Nakamoto Coefficient around like six or seven in those early days. And now it's 24 and that may sound like a small number, but when you look at Bitcoin, it takes about six mining pools to get to 51%, an Ethereum, I think three or maybe four, something like that mining pools, ETH2 stake distribution is even worse. It's like Lido, might be Lido plus one exchange, Kraken or Binance might cross it.

Anatoly Yakovenko (16:17):

And it's both important and not as important as people think. So the core part of where security comes from these networks is actually in the sheer number of nodes, because fundamentally like when high level catastrophic event happens, a bunch of machines get corrupted. As long as one of them can recover the data, that data contains cryptographic signatures from everyone else in the network. They can validate that that data's correct, and the network can recover.

Anatoly Yakovenko (16:47):

So you only need one out of many. And this is true about every BFD system, even Bitcoin, if all the Bitcoin ledgers are destroyed, doesn't matter how much cash power was securing it, Bitcoin is gone. So what's important is the number of copies of the Bitcoin ledger, and the same thing with ETH2 – it's a bit more complicated with ETH2 because they do the sub sampling thing, but effectively you need enough of the ETH2 validators to survive. So you can recover a full copy of the state.

Anatoly Yakovenko (17:17):

So that number is really, really what matters. And when you talk about the state concentration, Nakamoto Coefficient, that has impact on real time censorship. So if, for example, like Kraken and Lido decided to start dropping transactions, they could right now, you know, given the state distribution on ETH2, but the community can basically fork them all out. They can effectively tell them, because you're misbehaving.

Brian Friel (17:45):

You're breaking the social contract.

Anatoly Yakovenko (17:47):

Yeah, you're breaking the social contract. You can actually be kicked out. And that's, I think, kind of that action of last resort, it’s what really protects these networks. But given that you still want the Nakamoto Coefficient as large as possible, because I think, in the long term, especially for trading systems, you might not see this direct censorship. If the purpose of blockchain is to be these open marketplaces and the top three crypto exchanges can decide the order of events in the global open marketplace, there's probably perverse incentives there for them to not make this network the best marketplace in the world, because it's eating their lunch.

Brian Friel (18:28):

Yeah. We've reinvented payment for order flow here.

Anatoly Yakovenko (18:31):

Right. And that's really hard to enforce and may not even be intentional. Right? You may have somebody, some product manager at Coinbase that doesn't know what they're doing, that simply says, "Hey, I made these performance improvements to how we run nodes, and now our customers are earning more money." It may even not be like totally malicious. This is where I think that Nakamoto Coefficient actually matters in the long term.

Anatoly Yakovenko (18:59):

So Solana and Avalanche probably have the two highest, I forget what Avalanche is at, but basically it's much, much better than eight or three or four, whatever we see on Bitcoin and Ethereum these days.

Brian Friel (19:10):

So switching gears just real quick, we talked a lot about the network, but from an end user and from a validate perspective, one final piece of this is the developers who are actually building the applications. So the programs that run on this. You know, I mentioned earlier that Solana was kind of written off by a lot of people, especially in the ETH community, that there was no EDM-compatible, like native bridge over there, Chase Barker, lovingly calls, everyone who develops on Solana “Glass Eaters”. Has this kind of developer ecosystem that I view as very genuine to Solana, has this surprised you guys, do you foresee this continuing to grow separate from Ethereum? Are they going to merge, at some point remain specialized?

Anatoly Yakovenko (19:48):

It surprised me how quickly it happened. So me as an engineer, I don't like Solidity because it's not a well designed language. And the EVM is not a well designed virtual machine. And I have spent like a career working on virtual machines. So I get like an allergic reaction looking at its insides, and like, this is worse than like the first versions of Java, of the JVM.

Brian Friel (20:14):

Yeah. Hard to be inspired.

Anatoly Yakovenko (20:16):

So I kind of knew like when we made a bet on Rust, that was intentional and we didn't have enough funds to pursue like supporting VM, which I think we would've, if we did, and that probably saved us, because we had to pick like, what is actually core and important to Solana, and like, if we had to pick the one thing that we were building, it had to be differentiated from like, it had to actually showcase the most important thing about the network and that's performance. And you really, really can't write high performance code in Solidity or EVM.

Anatoly Yakovenko (20:49):

And Rust was just a, kind of a natural fit because it is a modern systems language with a lot of tools to build high performance software. So that was a bet that we made. And I knew that there's enough engineers out there that are like me that are naturally curious. They want to build a new stack. Oftentimes, like in my career, I would switch stacks every two years simply because I just got bored of the tools I was using.

Anatoly Yakovenko (21:15):

But I didn't realize, I didn't expect how quickly that would happen. And like how passionate people like became about it. Even though our initial version of the SDK was really, really bare bones. This is where this idea of eating glass came out of is that like, in those days you had to write everything by hand, all this deserialization calls by hand. There was like, the binary format was like unsafeCast. So you would do a cast of the data structure directly. There was no ideal and-

Brian Friel (21:46):

Wow. How far we've come with Armani and Anchor.

Anatoly Yakovenko (21:49):

Armani basically was like, "This is terrible. I'm either going to, like, I'm either going to quit or like, I need to fix it."

Brian Friel (21:58):

Or I'm going to put the whole ecosystem on my back, which he essentially did.

Anatoly Yakovenko (22:01):

Yeah. Yep. So he took on that role and I knew there was going to be one of those engineers eventually. This ... because I would've been that person, like, five years back in my career, I would've like, "F this. This is like terrible. I know how to fix it. So I'm just going to spend a week building the tools," but it was really, it would've been really, really hard for us to do it internally because it's easier to build those tools when you're building the product that's using them and not when you're like trying to supply them. I don't know if that makes sense.

Brian Friel (22:33):

You might be too close to it being, working on the protocol and not-

Anatoly Yakovenko (22:35):

Exactly.

Brian Friel (22:36):

... not like a consumer.

Anatoly Yakovenko (22:37):

So when you're like building the product, like an application, you know exactly what your needs are and you know what to prioritize and when I'm building the operating system, I'm just trying to be hypothetical. Well, the hypothetical application where the hypothetical engineer needs this and the requirements are just never right. Like it they're always wrong, basically.

Anatoly Yakovenko (22:57):

So, this is why I think it was really – the reason Anchor is successful is because Armani was building a product and he like, kind of knew exactly what he needed. And I think when looking at Anchor, I would've made totally different choices if I was building it. And I don't think it would've been as good in the end.

Brian Friel (23:14):

Oh, okay. I was going to say, has Armani heard that? But that's a nice compliment to throw in there at the end. Well, so wrapping up here a little bit, two last questions for you. One that we want to know here at Phantom is, what is the craziest idea or project that you've heard about that could be built on Solana?

Anatoly Yakovenko (23:32):

Serum, I mean is still definitely crazy to me, it's like a central limit order book that Jump is market making on, on a decentralized blockchain. That's pretty wild. I think there's a ton of games and payments and all this other stuff that's coming out.

Anatoly Yakovenko (23:46):

I think one of the wilder ones – there was the SolDate thing at one of the hackathons, which was a dating app. It didn't go anywhere. But people were like thinking outside of the box, which I thought was ...

Brian Friel (24:00):

Oh man.

Anatoly Yakovenko (24:03):

The games, I think like Aurory is like finally shipping, like workable parts of the game. I think that is really cool. And obviously like Star Atlas is such a huge ambitious project, that like, it's pretty wild to think that somebody's going to build from the ground up purely funded by NFT sales, like a AAA game, like at the level of EVE Online. So that, those are really, really ambitious.

Brian Friel (24:30):

And then our closing question here, who is a builder that you admire in the Solana ecosystem?

Anatoly Yakovenko (24:35):

So Armani obviously, but I want to like give a shout out to somebody else. I think, the Saber and the Mango guys. So Ian and Max and Daffy, I think have been just as important in like building a lot of tooling and onboarding devs, and you know, they're just awesome folks that are like constantly building things. So it's really cool to see them continuously shipping products. Even like when DeFi's up or DeFi's down, like when you look at the broader market, they're still building things and shipping things and that's kind of the most important thing.

Brian Friel (25:10):

Yeah. And that's Ian and Dylan from Saber, which is an automated market maker for stable pairs, and Mango Markets is a perps trading platform, a central limit order book in Solana.

Anatoly Yakovenko (25:20):

There's so many like the Orca team. When we first talked to them, they're like, "We want to build a really usable DEX for humans." And that was their whole vision. And you look at their daily active user numbers, it like shot up, I think more than Uniswap now.

Brian Friel (25:38):

Yeah. Oh, it's insane. Yeah. They're on a crazy growth trajectory now. I think that speaks to their UX. I will say just that one last note on the Saber brothers, there was a fable story of how they were sleeping in the Solana office at one point just shipping code. And I came in there and I saw for myself the mattresses in the meeting rooms there. I think we've all grown up since then.

Anatoly Yakovenko (26:01):

Yeah. A little bit.

Brian Friel (26:01):

That might be a thing of the past, but a little bit of lore for Solana development history there.

Anatoly Yakovenko (26:05):

Yeah.

Brian Friel (26:06):

That's great.

Anatoly Yakovenko (26:08):

When you're in that part of the office, you can still smell their spirit.

Brian Friel (26:14):

I love it. And it's totally, this is a really great discussion. Thanks so much for coming on the podcast. Everyone, very appreciative of what you built here at Solana.

Anatoly Yakovenko (26:23):

For sure.

Brian Friel (26:23):

So thank you for what you do.

Anatoly Yakovenko (26:25):

For sure. Thank you so much. Take care.