A Regression Task or Predict a Probability?

The goal of this bench mark is to predict a floating point value between one and five that indicates how similar two sentences are. What we ended up doing in that case is just quantizing the ground truth floating point values to the nearest point two. It does actually turn out to work well and achieve extremely competitive results on that bench mark.