On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

The paper and its question The paper is by Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell, and it appeared in the FAccT '21 proceedings, pages 610 to 623 ^[1]. It was published on 1 March 2021 with the DOI 10.1145/3442188.3445922 ^[1]. It appeared at the ACM Conference on Fairness, Accountability, and Transparency in 2021 ^[1]. The abstract notes that the previous three years of work in natural language processing had been characterised by the development and deployment of ever larger language models, especially for English ^[1]. It names BERT and its variants, GPT-2 and GPT-3, and most recently Switch-C ^[1]. Against this background, the authors step back and ask how big is too big, what risks are associated with the technology, and what paths are available for mitigating them ^[1].

How the argument is framed A language model is, in broad terms, a system that learns patterns from text and can be used to generate or assess language. The paper places its question inside a period in NLP when larger and larger models were being developed and deployed, especially for English ^[1]. The authors ask what risks come with the trend and what can be done about them ^[1]. Mitigation, in this setting, means the steps that could reduce or address those risks. The title's question, can language models be too big, is therefore tied to costs, consequences and choices, not just to size ^[1]. The paper's main move is to put size, risk and mitigation in the same frame ^[1]. It asks for the question of size to be considered alongside what the technology is for and what it may cost ^[1].

Recommendations The authors make several recommendations. One is to weigh the environmental and financial costs first ^[1]. Another is to invest resources into curating and carefully documenting datasets rather than ingesting everything on the web ^[1]. They also recommend pre-development exercises evaluating how the planned approach fits into research and development goals and supports stakeholder values ^[1]. Finally, they encourage research directions beyond ever larger language models ^[1]. Taken together, these recommendations cover costs, data practices, project planning and the direction of research ^[1]. They give the size question a practical shape: before building a larger model, one can ask what it will cost, what data it will use, what goal it serves, whose values it supports, and whether another research direction would be better ^[1]. Cost accounting asks developers to look at environmental and financial burdens before committing to a project ^[1]. Data work asks for curation and documentation instead of indiscriminate web ingestion ^[1]. Pre-development review asks teams to check alignment with research goals and stakeholder values ^[1]. The final recommendation widens the agenda to research that is not defined by ever larger models ^[1].

Publication and significance The paper appeared at FAccT in 2021 ^[1]. Its contribution is to connect the growth of language models to environmental and financial costs, data documentation, research goals, stakeholder values and alternative research directions ^[1]. It does not settle the question of size with a single number; it lays out a set of considerations for deciding how to proceed ^[1]. For readers, it offers checkpoints for discussing whether and how to build larger systems ^[1].