So you get a lot of advantage to that. Like, so let's take the example of a system call where like, I guess the flow of events is that your Go program tries to write to a file under the hood, the ghost and library is using the right system call leading up to the system call, your Go program puts whatever information it needs to in the correct registers. And then it executes a system call instruction. And then the kernel takes over executes the system call returns back to user space with, hey, we were able to write to that file or whatnot. In the eBPF world, you can write an eBPF program that is triggered by system calls. So if you were to have an eBPF program that is triggered every time the right system call is called, so adding to that original flow of events, the Go program sets up the system call and executes it right before it executes it, the eBPF program runs. It can look at all of the arguments that were passed to that system call. It can do whatever it wants and then once it's finished, the system call executes, then returns back to user space, et cetera. And that's the eBPF side is completely invisible to the application that actually triggered it. Interesting.

So does it

run as a kind of background to the main things that are going on? Or is it a blocking code? How does it actually run when it sees

a particular call has happened? So in the case of a probe attached to a function or something like that, it's blocking. So for that moment of time, the execution of the program that invoked or triggered the probe will be stopped so that the eBPF program can do any kind of inspection or whatever that it needs, which is also why for the context of tracing, this is what got me interested in eBPF was the combination of the two last subjects we were talking about, which is like the overhead of system calls and just the overhead of tracing in general. So a similar thing would happen. So the reason why I got interested in it was because I wanted to make delves the tracing back end more performant, like have less overhead so that maybe you could use it like in a production context if you wanted to. I did a tweet a while back when I first started working on it, the overhead added. So I had a program that ran in some odd number of microseconds. And then with eBPF based tracing on top of it, it went from like 20 microseconds to 300 microseconds or something like that, which sounds like a significant amount of overhead, but we're talking microseconds. And then I timed it using the P-Trace-based tracing that Delve had been using prior and it bumped up to like 2.3 seconds. So you go from microseconds to actual seconds, which is like you can't have that kind of overhead if you're tracing something in production. So that was kind of what got me interested in it. Having these really small targeted programs that can be called as a result of something that don't do context switching between kernel and user space and stop the program for as little time as possible and really get kind of performant detailed, but also ad hoc tracing. The other kind of difficult thing was like making it ad hoc. So a lot of times when people write eBPF programs, they're very, very targeted. You already know when you're writing the function what corresponding function it's going to be attached to, like what kernel function or whatever, like you generally already know or whatever. So you kind of know the arguments to expect and things like that, but in the case of Delve and what I was doing with eBPF, I was kind of abusing it a little bit to try to do like just I want to attach this probe to a completely arbitrary function where I don't know how many arguments or return arguments it has. I know nothing about it, but I want to get all the information out of it. How do I do that? And so that opened up a lot of questions of how to write a generic eBPF program and how to communicate between that program and Delve, right, like communicate between kernel space and user space in a way that also want to introduce back the slowness that I was trying to get rid of.