How Do You Deal With a Poison Message?

AWS has invested a lot in just trying to understand when APIs go down. When we see one starting to fail, we see if it's failing for a lot of users or if it's just one user. We'd email them, hey, heads up. Something funny is going on with your zap. But if it's something that's happening across a high cardinality of users, a bunch of different users are hitting this. Now we know, okay, something looks a little fishy here. And at that point, we'll start to hold things back and periodically like kind of let them through until they clear up.