This talk will be based on our recent paper accepted to ICLR 2021. 

Fine-tuning pre-trained Transformer-based language models such as BERT has become a common practice, dominating leaderboards across various NLP benchmarks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process—training the same model with multiple random seeds can result in a large variance of the task performance. Previous literature identified two potential reasons for the observed instability: catastrophic forgetting and the small size of the fine-tuning datasets. 

In our paper, we show that both hypotheses fail to explain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT, which are fine-tuned on three commonly used datasets from the GLUE benchmark, and show that the observed instability is caused by optimization difficulties that lead to vanishing gradients. Additionally, we show that the remaining variance of the downstream task performance can be attributed to differences in generalization, where fine-tuned models with the same training loss exhibit noticeably different test performance. 

Based on our analysis, we present a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the previously proposed approaches.


Maksym Andriushchenko is a second-year PhD student in computer science at EPFL (École Polytechnique Fédérale de Lausanne) in Switzerland. He obtained his MSc from Saarland University, Germany. His research mainly focuses on how to make machine learning algorithms adversarially robust and improve their reliability. Maksym has published seven papers at major machine learning and computer vision conferences (NeurIPS, ICLR, AISTATS, CVPR, and ECCV).