Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

Journal of Machine Learning Research 21(140), pages 1–67
2020

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new “Colossal Clean Crawled Corpus”, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

Publications
home    search    all

Publications / Views
tags  •  authors  •  editors  •  journals
2023  •  2022  •  2021  •  2020  •  2019  •  2018  •  2017  •  2016  •  2015  •  2014–1927
in journal  •  in proc  •  chapters  •  books  •  edited  •  spec issues  •  editorials  •  entries  •  manuals  •  tech reps  •  phd th  •  others
online  •  in press  •  proof  •  camera-ready  •  revised  •  accepted  •  revision  •  submitted  •  draft  •  note
ACM Digital Library  •  DBLP  •  IEEE Xplore  •  IRIS  •  PubMed  •  Google Scholar  •  Scopus  •  Semantic Scholar  •  Web of Science  •  DOI

Publication

— status

published

— sort

article in journal

who

— authors

Colin Raffel    Noam Shazeer    Adam Roberts    Katherine Lee    Sharan Narang    Michael Matena    Yanqi Zhou    Wei Li    Peter J. Liu

where & when

— publication date

2020

— journal

Journal of Machine Learning Research

— volume no.

21

— issue

140

— pages

1–67

URL

files
Open Access PDF

Partita IVA: 01131710376 — Copyright © 2008–2023 APICe@DISI – PRIVACY