So before,

the black wasn't quite as dark. Okay. just green screen yourself not quite all right so what's going on right now so something just went behind your head and it looks like grafana yes some sort of dashboard is behind your head gearhard yeah so

that's one of the early birthday presents which i couldn't wait to open yeah your birthday is coming right up isn't it it is yeah it is i think by the time this will this will be out it'll be out uh

all right if you're listening to this find gearhard tell him happy birthday thank

you i appreciate that so um i always wanted to have a big ass monitor, like really, really big. So big. A bam, as they call it. A bam. There you go. I always wanted to have a bam. Bam. Bam. It's got one. And that's what happened behind me. Like the whole screen, like the whole background is actually now a one giant screen. This

is a real screen back there?

Yeah. Is it a TV screen or is it a TV screen it's a bam

it's a bam tell us

more about this yeah i've heard of this

so what is it it's a it's a samsung s95d okay and it's a 65 inch tv so it's big and what it means is that I can talk to anyone and I can see exactly what's happening across every infrastructure. Right now I have the changelog infrastructure running there. And do you see those spikes right there? Do you know what that spike is? You just now. Yeah, exactly. That's me just now. I did that. I created that spike. I did that. That spike is me so proud of himself i did that spike yeah yeah you did so that spike right there is the benchmark that went directly to fly now on the left hand side as you as you look at it that's the all the metrics coming from the fly application our fly changelog application on the right it's all a honeycomb and because it's a bit blurry you can't see the details which is exactly what we would want right we don't want to advertise all the details but really what's interesting is the shape of it so honeycomb i can't figure out how to automatically refresh grafana in fly has that capability so i just need to manually refresh it i have to click the refresh so let me do that there you go refresh he's leaning over leaning over i'm hitting refresh and then you should see that other half refresh right like half of the background refreshed yes and actually it's the same time stamp so i need to go the last 24 hours there you go now we're looking at the last 24 hours so do you see those spikes there yes those spikes is the benchmark which i did against fastly against our cdn so you can see that we never hit those levels under normal operating conditions right that's like yeah 100x what we normally operate so maybe being able to serve 10 000 requests per second doesn't make that much difference since really

we never hit those those levels i feel like we've gone to target the three of us y'all took me to the toy department and you said pick a toy i chose my. We went to the checkout. Target is a popular store. Here, by the way, Garn, if you didn't know Target. We checked out. We successfully paid. We've left. We've gone home. And you've not given me my toy. Right. Where is my toy? Well, I

can't get the monitor for you. You need to get it for yourself. No, I mean, Pipely. Pipely. Pipely. It's the toy. So Pipely, if you go to cdn2.changedog it runs. It now uses a component that terminates TLS to origins, and now we need to add more origins. While it's half as fast as the current CDN is, we know that it can sustain all the load that we need to replace our cdn so the toy is the toy will the toy will work in terms of what comes next we need to configure more origins we need to get for example the feeds one the assets one and we need to scale the instances in such a way so they can handle the traffic right now we only serve we only save save the assets we only save the responses in the actual memory so we need to configure disk there's a couple more things a couple more knobs to configure but this is getting closer and closer

and closer i feel like the real toy is the samsung95D 65 inch OLED HDR Pro glare free with motion accelerator. Gerhard that sucker is expensive man. Well nice eBay half price. That's

what I say brand new. So you just need to shop around. Do what Adam does, you know? Okay. Do what Adam does, basically. Well, I was going to let you know, you know, my birthday is July 12th.

All right. Just in case you're wondering. Cool. Mine sooner. March 17th.

March 17th. I wasn't really jealous of all your computers you were talking about earlier. But now you are. That screen is amazing. Holy cow. Yeah. That is my screen. All right. So you scaled up our pipely to well the performance x1 performance 1x

it didn't work uh fail yeah it failed so maybe we're trying to scale too much let us down oh dang man it was cool until i didn't know i didn't know what would happen maybe we uh yeah let's see if you do fly ctl let me just do that let me go fly ctl machines list let's see what's going on uh live debugging why not so we see performance one two only two really but the rest could not be scaled and i don't know exactly why so let's just do that again let's do vm scale updating machine see this this other one just couldn't update it and i'm not sure why exactly that's the one in heath row waiting for machine to be happy sorry to be healthy to become happy a healthy machine a happy machine is a machine indeed. That's right. So that's still waiting for machine. Okay, so now it's moving to the next one.

How many pipeline instances are we running right now?

One, two, three, four, five, six, seven, eight, nine, ten. Ten. So there's ten

of them in different regions around the world. Yeah. And we've got two of the 10 upgraded to performance 1x the other ones are on shared cpu 1x exactly and which has also uh 10x the ram it looks like so the shared cpu is at 256 megabytes whereas the performance 1x is at 2048 exactly yeah

so

that's quite a scale

yeah it's it's about 10x and i'm wondering if we do that 10x like how will it behave and

what would

I mean, we can check it exactly to see how much that would cost. And maybe we don't need 10. Maybe we need just one per continent. Maybe that will be enough or one per like East Coast, West Coast. and like this is like you remember like the old one so there's a couple of optimizations which we can we can change there so what's the question how much will it cost well

i was just wondering how much extra it is we don't even get the exact answer okay these are just concerns that i have as we move forward and then is 10 even the right number is a question i mean maybe it's smarter to have to leave it at the shared cpu 1x but have 30 of them versus 10 at the performance 1x for instance yeah

i think so we can see that like the we went with the cheapest one right smallest one um so shared cpu 1x uh you get i think you get the bandwidth which depends on how much um how big the instance is right you get like a fair share of the bandwidth so these instances they were costing like less about two dollars per month um just in compute costs we went to performance 1x which is 31 so that is more than a 10x jump maybe if we went to a 4x sorry shared cpu 4x which is about eight that would have been a 4x yeah 4x and also like a more realistic more realistic upgrade but i i wanted to make sure that we get like um we get the higher tier ones performance 1x is like the lowest high tier one which means that you get a full core is not getting throttled and my assumption is you'll also get more bandwidth and that's what we're testing here if we go to like the next tier of instance which is like compute optimize in a way it's like a huge jump but does that translate to bandwidth performance so we're still going through that i mean we can try benchmarking it again to see how it behaves and the reason why you could benchmark it again is because um the pop the one in he throw has already scaled so let's see you know how this one compares we are pushing 480 470 480 okay so i think we'll get a similar result i think i

needed 10 million again 480

i did uh i think 100 000 requests sorry 100 000 requests in total yeah um just to throw some node its way and see how that behaves and we're 4 000. so apparently scaling scaling up the instance did not increase the bandwidth interesting so the question would be is this as much as we can get and should we could we go higher i don't know is

the limitation the network is that what we just resolve to them because cpu and other things didn't really influence it? RAM didn't influence it?

Yeah. So I would ask, for example, Fly, like how do they allocate network bandwidth based on instance size? Like how do those limits work?

Yeah, that's not clear.

And so that would be one question. And what I'm wondering, is 4,000 enough? Because we're looking at the graph we're seeing the spikes and apparently we we never even hit 4 000 requests per second on our existing cdn it means that the ceiling is lower but since we're never hitting that ceiling maybe that's okay not to mention that we've seen bunny for example this is a perspective which i haven't seen in bunny before where we can see the throttling kicking in right we can't even benchmark it properly because it throttles you much earlier and i looked through the config i went through the settings like cdns apparently they're not all configured the same which is why i was looking at varnish to see what can varnish do like where exactly is this bottleneck coming from and are we okay with the ceiling?

Is it necessary to have this throttling in place? For

who? For,

I guess, just the system, the uptime of the system. The vendors.

Yeah, it would make sense for

them too. I mean, for us, like, I mean, we would be at least temporarily not, Pipely would not have a lot of users, I would say. Like we would deploy pipely on-prem basically like it would not be a service we're consuming pipely would be software we deploy for us to use and so do we really need throttling if we're our own user and we control our systems oh i

see what you mean you

see what i'm saying like because bunny has it probably is a safeguard because they're public whereas pipely would be deployed for us in our use case right so we don't need throttling

no deployed for us but it'd be hit by randos around the world true so we can get dosed yeah

we could that

is that is a real possibility you send us 5 000 requests a second we're dosed to one pop at least to yeah yeah exactly yeah so i think some form of rudimentary throttling makes a lot of sense i don't think it would add very much in terms of software on our side to say you know you can make it very rudimentary this ip can only have so many requests

a second done i think then you're at least avoiding that low-hanging fruit i haven't looked into that but having this discussion is valuable i mean this is why you know like i don't think we can have the toy because we're still debating what the toy should be and how it should behave but it's exactly we're building our toy and i think this just makes it more real because these are the steps that we would go through before we take this toy into production and i think this is the perspective which is valuable to have right like the level of care and attention and detail that we go through to make sure that what we put out there will behave correctly and the comparison that we have right now is fastly which you know from some perspectives it behaves really well i we're seeing performance is amazing caching not so good but again it makes sense why they you know don't keep content in memory as for example we would because we would optimize for that which means that because we optimize for that we want to store as much of it in memory as possible memory that we pay for or right no discs that we pay for wherever they may be and then i think this will also um we'll have questions about like how should we size those instances we just heard that maybe the performance 1x maybe it's a bit too expensive because we need to run a bunch of them and how many if you remember first time we were running 16 maybe that's a bit too many maybe 10 is a better number but even that might be might be too much now if we're looking at the cost right we're paying 30 per instance and if we have um 10 of those we would be paying 300 per month for the compute i think i think that's okay i think i think that's not crazy in terms of

cost let me ask you a question. Maybe this is a stupid question, but let's ask it anyways. We want to store, are we designing the system to be memory heavy, where we have terabytes of memory available to the system, so we can store all of our data in memory? I don't think so. Or just on disk and have lots of memory available if we need it so

i think that we need both i think that the data which is hot should reside in memory and think about how zfs works the file system and when you have an arc so this would be exactly that

yeah we

would want to store the most often data in memory and the one which is least accessed on disk so i think we need both because the memory we can scale it i mean if we go back to two gigabytes of memory right for ten dollars let's say we go we keep the 1x we can get eight gigabytes of memory that doesn't seem a lot of memory for example i wouldn't know and this is like where the cash statistics would come in handy how much data do we frequently serve and i know that we have the peaks right when we release something there's like a bulk of content that we serve often how much is the bulk of how much is the hot content i don't have an answer to that but all these things are getting us closer to those concerns shall i say that the system will need to take care of honestly i don't think that we should give it more than for example 16 gigs per instance and even that might be a bit big and i'm wondering whether all regions should have the same configuration and i'm thinking no because maybe in south america and i know this for a fact there's less traffic than for example in north america and maybe oceania i'm sorry asia let's go with asia again it's less traffic than we have in north america and even europe so then i think the same configuration across all regions doesn't make sense but knowing how much data is hot i think i think that's something important how

would we know that just based on like stats to the direct

stats itself yeah stats from the cache to see how much of those like how much cache is being used and are there any configurations and i haven't even looked into this are there any configuration in terms of evictions like how frequently should we automatically drop content? I think this is where our cache hit ratio will come into play, right? So if you don't store enough of it in memory, you will have a lower cache hit ratio. While if you store too much, maybe you're being wasteful. I mean, having a high cache ratio, cache hit ratio, while a lot of that data is infrequently used, you're paying for memory that you don't need the other thing the other question which i have is are the nvme disks fast enough and if we think about netflix netflix does the same thing right they put their those big servers in isps they cache the content on those big servers so that they can deliver them really quickly to customers wherever they may be right we're not going to go there right so so this is not that but that's one pattern that they apply because they realize the importance of you know having lots of content close to users memory is not big enough you need disks again we're not there we don't have that problem well

we're getting there i think we have some decisions to make as we go i think roughly speaking the dog hunts i think 4 000 requests per second well managed uh will be fine yeah and i think we'll find out otherwise and be able to scale one way or the other um around such issues what else is left in pipely's roadmap you know as we look towards the future now because yeah let's do next steps so i think that now we

are finding the place where we can add feeds back feed feeds backend and we also need the um static assets one so i would add both when we add them we need to figure out do we store all that in memory and i think the answer is no because especially static assets they'll use a lot of memory but maybe disk and i think we should we should look into that can we configure different backends like how does that work we're basically getting like to the hard part of configuring varnish for our various backends and each back end needs to have a different behavior i think so that's something to look into logs sending logs to honeycomb i think that is a much easier problem to solve because we would be using vector and now we have the building blocks which we have the first sidecar process if you want to think about that if you want to think about it like that which means that we have there's varnish and there's a couple of other smaller processes that support it we have tls exterminator that terminates tls to or to origins to back ends the second one would be this um in my mind it will be vector.dev which is what we would use for these logs so vector.dev would get the logs from varnish and send them to honeycomb so it's an integration which i've used before i know how it works it's very performant it's fairly easy to configure and then would have another helping process that would work in combination with varnish to accomplish a certain task and honeycomb and s3 like all those it supports multiple sinks so collecting the logs on one side and just sending them like in multiple sinks that is very straightforward because it just handles all that itself and then really the last the last hard bit is purge across all application instances and i think that one is like maybe a step too far to think about it now but i think the way we so first of all now we automatically, so we have an image to publish. We are deploying the application automatically through CI. That's like some plumbing that you want to have in place. We have support for TLS backends, and that was an important one, especially when it comes to other origins, right? Because let's say if we are running in Fly, we can connect we can use the private network to connect to the changelog instance but for external origins like feeds we would need to go to http because we didn't have https so now we have https i think that's that's also like an important building block and now we're hitting the these there's like benchmarking i won't say we got sidetracked by it but i think it's like something worth considering because you may end up building something that won't work we won't be able to use this to replace our current cdn and the goal is to be able to say with confidence that pipely is able to do the work that currently our cdn is doing and what does that mean from a configuration perspective, from resources perspective? I think everything adds up and it feels like, it feels like we're more than halfway there. For real. I don't mean like, will this work? No, no, we're more than halfway there to replace Fastly with

Pipely for us. All right, take us to the promised land. Give us that toy. It

won't be christmas it'll be before christmas adam when is your

birthday march march okay oh that's too soon that's too soon jared yours is july okay all i want for my birthday is pipely and a samsung

s95d All right. Well, I behaved very well, I think. And my wife must love me very much because it was, yeah, it was a present from her to me. So, yeah. So she loves the nerd in me. She has to, she has no choice. They come as a package. If

she doesn't love the nerd, I mean, what's left? Well, that's what you

see, Jared, but this is on the show for that.

Don't answer that. Well, that's cool. I love this exploration though. I like that there's possibility to run your own thing like this, you know, to configure it to the way we want to. I mean, to zoom out, we, the has been that it's been hard to configure Fastly, not as a CDN, but as a CDN we need. Our particular use case is not that Fastly is not good as a CDN. It's just it has not been highly configurable by us. It's been challenging over the years, mainly just because they're, I think, designed for different customer types. We're a different customer type. And we've been holding it not so much wrong, but it's just been a square peg around a hole kind of thing where it's not perfect for our particular CDN needs. I think we've had lots of cache misses over the years. Like, why is our stuff not cached? Yeah. It seems like it should be. You know, we're not prioritized as a you know, as a thing to serve because of the way the system works. You know, and that's just it. And we're designing something that serves that kind of system. Where it serves the data. Holds more memory. Has more available to it. It is not a mess. That's right. Which I think is cool. Very cool. this tooling you build is so cool man i can't believe how how cool

this stuff is that you built man it's

really awesome thank

you it's coming together it's uh and i love the tv too yeah it is like a well-rounded experience right so the idea is to be able to have the tv on now it's a bit bright and it's it's it's running a little bit hot i can i can feel it it's it's not it's not uh uh winter yet i don't need heating in the office uh well it's like the end of the winter but um i'm i'm able to see a lot of metrics i think that's something that um i always loved to be able to see how things behave and when they misbehave to be able to see and understand at a glance what is wrong are getting ddos am i running out of memory like which instances is is problematic and i think this is just like a starting point i literally threw the two dashboards that we have over there but i haven't optimized them in in any way and i think having something like this just makes it in a more i don't like breathing living system well

yeah you can see in real time like what's happening the metrics is the life you know it's each you know organ so to speak your own knock yeah yeah

as

it is super cool it's super cool i am jealous uh i want one i inspired you uh

the christmas is far away right people can save uh i know that uh we did and uh it's been in the making for a long time so before before i could get this i had to get a system that is able to power it that's what pop os does one of the things which it does you know it has a gpu right which is powerful enough to be able to to power it i have another monitor here which does like screaming rings so i can change things here and set things up a system that's able to be on and that it's not too loud that was like another consideration a black wall so it blends nicely it was like so many things like years in the making as pipely is years in the making that's right and it'll be just as beautiful as that i

would love a uh tour a home lab tour i would love that coming

soon to well i'm working towards that uh but i can i can show you one more thing which was not planned i know we're a bit over time but if you want to see one more thing i'll show you my m25 every year i basically uh take one of these machines online and i have like a 24 25 so this is this year this is the machine that that came online it's a it's running through nas as you can see it's an i9 990k it has 128 gigs of ram so it's like fully maxed out basically and this is how just move that screen a little bit uh this is um you can see all the storage has two pools it has an ssd pool and an hdd pool um so spinning disks and some slower ssds um they're the evo i think 870 and um it's something that you need to have in place to be able to have decent storage between linux and windows and mac and everything to just work so that was one of the projects and i didn't have time to talk about it maybe maybe next time i know that you are a true nas user adam zfs and like all like all that stuff so yeah

several polls several things similar to this slightly beefier machine it's a a xeon processor and it's a 42 10 i want to say silver yeah i think there's a hundred and some gigs of ram i want to say 192 maybe 128 okay it's something like that it's not 256 i know that for sure right i don't have a need for it i mean it's it's nice it's a tinker's dream to have lots of ram and a zfs system but i just don't need it it's just you know i caught myself just wanting to have it just to have it and i'm like yeah that doesn't make any sense you know like you spend all the money on the ram just spending the discs instead yeah you know because discs are expensive yeah

for video storage you know something that that you will you would need be sure to do editing and especially if you edit it from multiple machines you need like a fast network a couple of other things but yeah

my home lab's suffering right now i don't have 10 gigabit everywhere i do have it in the network i just don't have it everywhere. So I'm in the process of fixing that. There's some slight life updates for me that will make it more important, I should just say. I see. I had a flood in the studio. Oh, wow. Okay. I don't think I can stay here anymore, let's just say. I got to go home. Okay. I'm turning my home office into a true home lab and work lab and that's in the making so techno

it's a bummer and more techno team yeah

you know uh i mean i'll i'll be close to the things i play with more frequently i feel like i've always been like two location and it's been challenging because like right now i can't access true nas it's at home i can't access that windows pc it's at home it's in the home lab you know yeah um and i've just sort of like stripped away more and more here to the point point that it's like it doesn't make

sense to stay here any longer well your background will change it's a pretty