What's the Role of the Two Timestamps on the Server?

If the clocks were in sync, you know that the timestamp that you took minus the date I sent should be equal to the delay of the packet to get to you. And vice versa. So if you ask me what time it is, I write down when I receive it and then I write down the time where I send it back. You could imagine a simple protocol with just three timestamps. It certainly seems like something you can in practice ignore because if you just look at the number of parts per million or whatever that they were talking about in terms of how much drift there is in a real computer clock, I think that is in fact pretty tiny.