The Importance of Quality Metrics in Natural Language Generation

In natural language generation, there are these criteria like fluency, redundancy and coherence. But when we try to assist the quality of these automatic metrics, we usually talk about how well they correlate with human judgements. So, if the metric can actually measure at least some aspect of the humans we care about beyond N-gram overlap, overlap, I would say this metric is high quality. For instance, maybe a metric like Meteor, in addition to N-gram similarities, actually looks into some contextual similarity, which in some cases performs much better than N-gram metrics.