Abstract

Generating text with autoregressive language models (LMs) is of great importance to many natural language processing (NLP) applications. Previous solutions for this task often produce text that contains degenerative expressions (Welleck et al. 2020) or lacks semantic consistency (Basu et al. 2021). Recently, Su et al. 2022b introduced a new decoding method, contrastive search, based on the isotropic representation space of the language model and obtained new state of the art on various benchmarks. In addition, Su et al. 2022b argued that the representations of autoregressive LMs (e.g. GPT-2) are intrinsically anisotropic which is also shared by previous studies (Ethayarajh 2019). Therefore, to ensure the language model follows an isotropic distribution, Su et al. 2022b proposed a contrastive learning scheme, i.e. SimCTG, which calibrates the language model’s representations through additional training.