Authors
Sidi Lu, Hongyi Liu, Asli Celikyilmaz, Tianlu Wang, and Nanyun Peng
Overview
Contrastive Distribution Methods, or CDM, use the gap between a stronger and a weaker language model as a signal for text quality. The framework provides a reference-free alternative for evaluating open-ended generation.
Generative CDM uses the distributional contrast to create negative examples for training discriminator-based metrics. Discriminative CDM directly aggregates differences between the two models’ token-level probabilities into an instance-level score.
Results
Experiments on multi-turn dialogue coherence and commonsense evaluation for controllable generation show that CDM correlates with human judgments more strongly than existing automatic metrics across the evaluated datasets.
Paper
Status
ICML 2024 Poster.
Citation
Sidi Lu, Hongyi Liu, Asli Celikyilmaz, Tianlu Wang, and Nanyun Peng. “Open-Domain Text Evaluation via Contrastive Distribution Methods.” Proceedings of the 41st International Conference on Machine Learning, PMLR 235:33057–33068, 2024.