Regularizing transformers with deep probabilistic layers Articles uri icon

publication date

  • April 2023


  • 161

International Standard Serial Number (ISSN)

  • 0893-6080

Electronic International Standard Serial Number (EISSN)

  • 1879-2782


  • Language models (LM) have grown non-stop in the last decade, from sequence-to-sequence architectures to attention-based Transformers. However, regularization is not deeply studied in those structures. In this work, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer. We study its advantages regarding the depth where it is placed and prove its effectiveness in several scenarios. Experimental result demonstrates that the inclusion of deep generative models within Transformer-based architectures such as BERT, RoBERTa, or XLM-R can bring more versatile models, able to generalize better and achieve improved imputation score in tasks such as SST-2 and TREC or even impute missing/noisy words with richer text.


  • deep learning; missing data; natural language processing; regularization; transformers; variational auto-encoder