Galactica

What it is

Galactica is a large language model aimed at scientific work. Meta AI and Papers with Code unveiled it on 15 November 2022 with a public demo. It is a text model that can summarise papers, solve math problems, generate wiki-style articles, write scientific code and annotate molecules and proteins, according to Meta's description at launch.^[1]

Origins

The paper, *Galactica: A Large Language Model for Science*, had nine authors, and the first author was Ross Taylor. It was submitted to arXiv on 16 November 2022.^[2] The code is in the paperswithcode/galai repository on GitHub, under an Apache-2.0 code license, with a separate license for the models.^[3]

The paper's title, A Large Language Model for Science, names that focus.^[2] LaTeX, the markup language many scientists use to typeset equations, was one of the tasks in the paper's reported results.^[2]

Main features

Training corpus. Galactica was trained on a large scientific corpus of 48 million examples. The examples include scientific articles, websites, textbooks, lecture notes and encyclopedias.^[1]

Released sizes. The released models were the mini at 125 million parameters, the base at 1.3 billion, the standard at 6.7 billion, the large at 30 billion and the huge at 120 billion.^[3]

Reported results. A benchmark is a standard test set used to compare models, and a higher percentage means a better score on that test. PubMedQA and MedMCQA are tests of medical question answering. The paper reported 68.2% on LaTeX equations, compared with 49.0% for GPT-3. It reported 41.3% on the mathematical version of MMLU, compared with 35.7% for Chinchilla, and 20.4% on MATH, compared with 8.8% for PaLM 540B.^[2] It also reported 77.6% on PubMedQA and 52.9% on the MedMCQA development set, which the authors called new state-of-the-art results.^[2]

What it was meant to do. Meta said the model could help scientists by summarising papers, solving math problems, drafting articles and writing code, and by annotating molecules and proteins.^[1]

Its place in development

Users quickly found the model could generate fake papers, sometimes attributed to real authors. It also produced wiki articles on invented subjects, such as the history of bears in space.^[1] Meta took the public demo down on 17 November 2022, about three days after launch. The models stayed available to researchers.^[1]

Yann LeCun, Meta's chief AI scientist, defended the model at launch. He later wrote that it was no longer possible to have fun by casually misusing it.^[1]

Galactica is a useful case for readers because it shows the gap between a model that sounds authoritative and one that is reliably correct. Its fabricated papers and articles came from the same generation ability that made its scientific summaries and answers useful, and the demo was the point where that gap became public.^[1]