--- generator: contextjamming-llm-context title: "The Ghost in the Machine · GPT-3 Oral History" description: "The oral history of the paper that started the Scaling Era: fourteen voices from inside OpenAI recount how \"Language Models are Few-Shot Learners\" shattered AI orthodoxy and turned scaling into industrial science." url: https://www.contextjamming.com/FewShotLearners-Oral-History markdown_url: https://www.contextjamming.com/FewShotLearners-Oral-History.md section: "Essays & Guides" content_type: "Article" author: "Bret Kerr" publisher: "Context Jamming (ACRA Insight LLC)" date_published: 2026-07-18 language: en-US word_count: 4058 image: https://pub-bcdb3a688294470c8056f904d7e6236b.r2.dev/GPT-3/GPT32.png --- > **The Ghost in the Machine · GPT-3 Oral History** — Essays & Guides, Context Jamming. This is a plain-Markdown copy of https://www.contextjamming.com/FewShotLearners-Oral-History, prepared for LLM context windows, agents and retrieval. > > Cite as: Bret Kerr, "The Ghost in the Machine · GPT-3 Oral History," *Context Jamming*, July 18, 2026, https://www.contextjamming.com/FewShotLearners-Oral-History ![Not Vanity Fair, July 2026 — The Ghost in the Machine: An Oral History of GPT-3](https://pub-bcdb3a688294470c8056f904d7e6236b.r2.dev/GPT-3/GPT32.png) *Watch the 90-second briefing[Download PDF](https://www.contextjamming.com/FewShotLearners-Oral-History/the-ghost-in-the-machine-gpt3-oral-history.pdf)An Oral History · Context JammingScroll* Watch first · 1:30 briefing ## Why this paper mattered Before the fourteen voices: what *Language Models are Few-Shot Learners* changed, from one-model-per-task to the Scaling Era, in ninety seconds. Read the narration 1. May 28, 2020. The world is locked down. A seventy-two-page paper appears on arXiv. 2. Until then, AI was a craft. One task, one model, thousands of labeled examples. Every single time. 3. A small team at OpenAI made a different bet: that performance climbs a smooth power law as you add compute, data, and parameters. 4. GPT-3 had 175 billion parameters. Ten times larger than any dense language model before it. 5. Then, the ghost in the machine. Show it a few examples in the prompt, and it continues the pattern. No retraining. They called it in-context learning. 6. Skills appeared with scale. Flawless two-digit addition. News articles readers caught only 52% of the time. 7. It shipped as an API. Several of its authors left to found Anthropic. And two years later, the recipe became ChatGPT. 8. This is the story of the paper that started the Scaling Era, told by the people who built it. Animated frame by frame in Python (PIL + NumPy). Narration is a synthetic voice (Kokoro-82M · af\_heart); the score is original. Figures from Brown et al. (2020) and Kaplan et al. (2020); the scaling curves are illustrative. [Captions (.srt)](https://www.contextjamming.com/FewShotLearners-Oral-History/film.srt) [Companion explainer: In-Context Learning at Scale](https://www.contextjamming.com/gpt3-in-context-learning) An Oral History · Context Jamming # The Ghost in the Machine An Oral History of GPT-3 and the Dawn of the Scaling Era ~11 min read · 2,524 words --- In the late spring of 2020, as the physical world ground to a halt under the shadow of a global pandemic, a different kind of awakening was taking place inside a climate-controlled server farm. In a silicon cathedral built by Microsoft, thousands of NVIDIA V100 GPUs were humming in unison, burning through megawatts of power and millions of dollars to run a single, monolithic mathematical equation. The project was code-named GPT-3. When the resulting 72-page paper, "Language Models are Few-Shot Learners," dropped on the arXiv preprint server on May 28, 2020, it shattered the prevailing orthodoxies of computer science. At 175 billion parameters, it was ten times larger than any non-sparse language model that had come before it. But its staggering size was only a symptom of its true breakthrough. The paper proved that neural networks, if scaled massively, ceased to be narrow, single-task tools and instead developed emergent, generalized abilities. They could translate languages, write poetry, and perform arithmetic—all without task-specific training. This is the oral history of how a small, ideologically driven team at OpenAI turned artificial intelligence from an unpredictable art into an industrial science. Brown et al., “Language Models are Few-Shot Learners,” [arXiv:2005.14165](https://arxiv.org/abs/2005.14165) ## More ways in Audio Overview·Generated by Gemini The Big Blob of Compute NotebookLM audio overview of this oral history 00:00 / 00:00 Watch · GPT-3 playlist *Video companionThe Ghost in the Machine · GPT-3 oral history* ![The OpenAI research team, 2020](https://pub-bcdb3a688294470c8056f904d7e6236b.r2.dev/photos/open-ai-research-2020-master.jpg) *Fig. ·From left to right: Daniela Amodei, Jack Clark, Dario Amodei, Jeff Wu (technical staff member), Greg Brockman, Alec Radford (technical language team lead), Christine Payne (technical staff member), Ilya Sutskever, and Chris Berner (head of infrastructure).* [Photo: Christie Hemm Klok](https://www.technologyreview.com/2020/02/17/844721/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/) Chapter 01Late 2019 – Early 2020 ## The Church of Scaling Before 2020, the artificial intelligence industry operated under a paradigm of hyper-specialization. The standard recipe was to pre-train a model on a large corpus of text, then explicitly "fine-tune" its weights on thousands of labeled examples for each specific task. This required a new model and a new bespoke dataset for every single problem. But deep within OpenAI, a splinter ideology was forming. A faction of [former theoretical physicists](https://www.contextjamming.com/architectural-determinism) and rogue engineers believed the field was overthinking the problem. They hypothesized that algorithms did not need to be cleverer — they just needed to be unfathomably larger. > *[Dario Amodei](https://contextjamming.com/founder-files/dario)*VP of Research, OpenAI > > I've had the same hypothesis since 2017. I wrote a document back then called "[The Big Blob of Compute Hypothesis](https://www.dwarkesh.com/p/dario-amodei-2)." The premise was simple: all the cleverness, all the bespoke architectural techniques, all the "we need new methods" stuff—it doesn't really matter that much. What really matters are just a few things: raw compute, data quantity, data quality, and training duration. You get the obstacles out of the models' way, and they just want to learn. If you scale up the reagents linearly, the reaction proceeds. > > [Share on X](https://twitter.com/intent/tweet?text=%22I%27ve%20had%20the%20same%20hypothesis%20since%202017.%20I%20wrote%20a%20document%20back%20then%20called%20%22The%20Big%20Blob%20of%20Compute%20Hypothesis.%22%20Th%E2%80%A6%22%0A%E2%80%94%20Dario%20Amodei%2C%20VP%20of%20Research%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-dario-blob)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-dario-blob) > > *[Ilya Sutskever](https://contextjamming.com/founder-files/sutskever)*Co-founder & Chief Scientist, OpenAI > > Up until 2020, from 2012 to 2020, it was the age of research. But then the scaling insight arrived. The big breakthrough of pre-training was the realization that this recipe is good. You say, "Hey, if you mix some compute with some data into a neural net of a certain size, you will get results." You know that you'll be better if you just scale the recipe up. Companies love this because it gives you a very low-risk way of investing your resources. > > [Share on X](https://twitter.com/intent/tweet?text=%22Up%20until%202020%2C%20from%202012%20to%202020%2C%20it%20was%20the%20age%20of%20research.%20But%20then%20the%20scali%E2%80%A6%22%0A%E2%80%94%20Ilya%20Sutskever%20(%40ilyasut)%2C%20Co-founder%20%26%20Chief%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ilya-age-of-research)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ilya-age-of-research) > > *[Alec Radford](https://www.contextjamming.com/founder-files/alec-radford)*Research Scientist, OpenAI > > In the past, language models were seen as novelty toys that could only generate a sentence that made sense once in a while, and only then if you really squinted. With GPT-1 and GPT-2, we started to see that language modeling implicitly trains a network to perform multiple tasks, because different linguistic structures require different types of reasoning. But the question was: how far could we push it? > > [Share on X](https://twitter.com/intent/tweet?text=%22In%20the%20past%2C%20language%20models%20were%20seen%20as%20novelty%20toys%20that%20could%20only%20generate%20a%20sentence%20that%20made%E2%80%A6%22%0A%E2%80%94%20Alec%20Radford%20(%40AlecRad)%2C%20Research%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-alec-novelty)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-alec-novelty) > > *[Sam McCandlish](https://www.contextjamming.com/founder-files/sam-mccandlish)*Research Scientist, OpenAI > > The idea that you could just make a model bigger and it would automatically get smarter was still heavily debated in the broader AI community. Critics argued you couldn't get semantics from syntax, that a model just predicting the next word would eventually hit a wall of diminishing returns. We needed mathematical proof that this wasn't just a heuristic, but a law of nature. > > Now Chief Architect at Anthropic — oversees pre-training, large-scale model training, and the RL infrastructure behind Constitutional AI, reporting to President Daniela Amodei. [→](https://www.contextjamming.com/founder-files/sam-mccandlish) > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20idea%20that%20you%20could%20just%20make%20a%20model%20bigger%20and%20it%20would%20automatically%20get%20smarter%20was%20still%20heavily%20deb%E2%80%A6%22%0A%E2%80%94%20Sam%20McCandlish%2C%20Research%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sam-m-debate)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sam-m-debate) > In January 2020, Jared Kaplan, Sam McCandlish, and a team at OpenAI published "Scaling Laws for Neural Language Models". Through rigorous empirical testing on smaller models, they discovered that an AI's loss decreased predictably as a power-law function of compute, dataset size, and parameter count. > *[Jared Kaplan](https://contextjamming.com/founder-files/kaplan)*Research Scientist, OpenAI > > We studied empirical scaling laws for language model performance on the cross-entropy loss. We found that the loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with trends spanning more than seven orders of magnitude. Other architectural details, like network width or depth, had minimal effects within a wide range. It was a stunningly clean mathematical relationship. > > [Share on X](https://twitter.com/intent/tweet?text=%22We%20studied%20empirical%20scaling%20laws%20for%20language%20model%20performance%20on%20the%20cross-entropy%20loss.%20We%20found%20that%E2%80%A6%22%0A%E2%80%94%20Jared%20Kaplan%2C%20Research%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jared-scaling-laws)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jared-scaling-laws) > [Interactive CompanionScaling Laws — Interactive ExplainerManipulate the N/D/C power-law frontier from the paper this chapter describes.Launch the explainer →](https://www.contextjamming.com/scalinglaws) The scaling laws dictated that for a given compute budget, the optimal strategy was to heavily favor model size over data volume. Optimal model size should scale proportionally to C^0.73, while data should scale only to C^0.27. In practical terms, if a lab acquired 10x more compute, they should make the model 5.5x bigger but only add 1.8x more training data. > *Sam Altman*CEO, OpenAI > > The scaling laws gave us the confidence to place a massive bet. If the math held up, we knew exactly what a 175-billion parameter model would yield before we even turned on the machines. The basic logic was clear. GPT-1 cost approximately nothing to train. GPT-2 cost $40,000. We knew GPT-3 would cost millions. > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20scaling%20laws%20gave%20us%20the%20confidence%20to%20place%20a%20massive%20bet.%20If%20the%20math%20held%20up%2C%20we%20knew%20exactly%20what%20a%20175-billion%20parameter%20m%E2%80%A6%22%0A%E2%80%94%20Sam%20Altman%2C%20CEO%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sam-a-bet)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sam-a-bet) > > *[Ilya Sutskever](https://contextjamming.com/founder-files/sutskever)*Co-founder & Chief Scientist, OpenAI > > The neural network is really about learning. Its entire being is about learning representations, and better representations lead to better performance. Performance on a wide variety of tasks gets better smoothly, predictably, as you make the neural network bigger, train it on more data, and train it for longer. We were seeing the scaling laws holding. I didn't see a reason why they'd stop. It is abundantly clear that just scaling up the existing neural network paradigm is going to lead to AGI. > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20neural%20network%20is%20really%20about%20learning.%20Its%20entire%20being%20is%20about%20learning%20representati%E2%80%A6%22%0A%E2%80%94%20Ilya%20Sutskever%20(%40ilyasut)%2C%20Co-founder%20%26%20Chief%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ilya-agi)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ilya-agi) > Chapter 022020 ## The Infrastructure Nightmare & The Ghost in the Machine Training a 175-billion-parameter model is not merely a software challenge; it is a monumental feat of physical engineering. OpenAI's exclusive partnership with Microsoft—cemented by a $1 billion investment in 2019—provided access to a custom-built Azure supercomputer that ranked among the top five fastest in the world. The system comprised more than 285,000 CPU cores and 10,000 NVIDIA V100 Tensor Core GPUs, all united by a 400 gigabits-per-second InfiniBand network. > *[Tom B. Brown](https://www.contextjamming.com/founder-files/tom-brown)*Lead Engineer, OpenAI > > I tried to build up the courage to switch to AI research. At the time, I thought, "Okay, it seems like sometime in our lifetimes we might end up making transformative AI... but I also got a B-minus in linear algebra in college." I was really struggling as a software engineer initially. But engineering at this scale becomes less about pure math and more about managing catastrophic, distributed failure. > > [Share on X](https://twitter.com/intent/tweet?text=%22I%20tried%20to%20build%20up%20the%20courage%20to%20switch%20to%20AI%20research.%20At%20the%20time%2C%20I%20thought%2C%20%22Okay%2C%20it%20seems%20like%E2%80%A6%22%0A%E2%80%94%20Tom%20B.%20Brown%20(%40NotTomBrown)%2C%20Lead%20Engineer%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-tom-courage)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-tom-courage) > > *[Benjamin Mann](https://www.contextjamming.com/founder-files/benjamin-mann)*Research Scientist, OpenAI > > To train the larger models without running out of memory, we used a mixture of model parallelism within each matrix multiply and model parallelism across the layers of the network. You are slicing the brain of this AI into pieces, distributing it across thousands of physical chips, and requiring them to communicate synchronously. > > [Share on X](https://twitter.com/intent/tweet?text=%22To%20train%20the%20larger%20models%20without%20running%20out%20of%20memory%2C%20we%20used%20a%20mixture%20of%20model%20parallelism%20within%20eac%E2%80%A6%22%0A%E2%80%94%20Benjamin%20Mann%2C%20Research%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ben-parallelism)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ben-parallelism) > > *[Christopher Berner](https://www.contextjamming.com/founder-files/berner)*Head of Infrastructure, OpenAI > > An efficient infrastructure for running things at that size is really critical. You have to deal with failures that are happening on a regular basis. Every few days, or even more frequently at large scale, something's going to fail. A server fails, or a network link starts to flap. You've got to diagnose it and fix it very quickly. We used Kubernetes as a batch scheduling system to dynamically scale our clusters, but the sheer physics of networking 10,000 GPUs via InfiniBand is an entirely different beast. > > [Share on X](https://twitter.com/intent/tweet?text=%22An%20efficient%20infrastructure%20for%20running%20things%20at%20that%20size%20is%20really%20critical.%20You%20have%20to%20deal%20with%20%E2%80%A6%22%0A%E2%80%94%20Christopher%20Berner%2C%20Head%20of%20Infrastructure%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-chris-infra)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-chris-infra) > The team trained eight different model sizes to track the scaling laws. TABLE 1 *Table 1: The GPT-3 Model Family Architecture* | Model Name | Parameters | Layers | Bottleneck (dmodel) | Attention Heads | Batch Size (Tokens) | | --- | --- | --- | --- | --- | --- | | GPT-3 Small | 125M | 12 | 768 | 12 | 0.5M | | GPT-3 Medium | 350M | 24 | 1024 | 16 | 0.5M | | GPT-3 Large | 760M | 24 | 1536 | 16 | 0.5M | | GPT-3 XL | 1.3B | 24 | 2048 | 24 | 1M | | GPT-3 2.7B | 2.7B | 32 | 2560 | 32 | 1M | | GPT-3 6.7B | 6.7B | 32 | 4096 | 32 | 2M | | GPT-3 13B | 13.0B | 40 | 5140 | 40 | 2M | | GPT-3 175B | 175.0B | 96 | 12288 | 96 | 3.2M | All models used a context window of 2048 tokens. As the 175B model ingested its 300 billion training tokens, a strange behavior emerged. The researchers noticed they didn't need to update the model's weights to teach it a new task. If they simply provided a text prompt containing a few examples, the model would continue the pattern. They called this "In-Context Learning". > *[Alec Radford](https://www.contextjamming.com/founder-files/alec-radford)*Research Scientist, OpenAI > > Even before fine-tuning, generative pre-trained models display a baseline ability to perform downstream tasks simply by conditioning on the right text. But scaling from 1.5 billion to 175 billion parameters unlocked abilities that were entirely absent in smaller versions. The model acts as a meta-learner. > > [Share on X](https://twitter.com/intent/tweet?text=%22Even%20before%20fine-tuning%2C%20generative%20pre-trained%20models%20display%20a%20baseline%20ability%20to%20perform%20downstream%E2%80%A6%22%0A%E2%80%94%20Alec%20Radford%20(%40AlecRad)%2C%20Research%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-alec-meta)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-alec-meta) > > *[Tom B. Brown](https://www.contextjamming.com/founder-files/tom-brown)*Lead Engineer, OpenAI > > The traditional fine-tuning paradigm requires thousands of task-specific examples. But humans don't learn that way. You can give a human a brief directive in natural language, or at most a tiny number of demonstrations, and they can perform a new task. We wanted to see if scaling up language models could achieve this same fluidity. > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20traditional%20fine-tuning%20paradigm%20requires%20thousands%20of%20task-specific%20examples.%20But%20humans%20don%27t%20lea%E2%80%A6%22%0A%E2%80%94%20Tom%20B.%20Brown%20(%40NotTomBrown)%2C%20Lead%20Engineer%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-tom-humans)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-tom-humans) > The paper formalized three settings: Zero-Shot, One-Shot, and Few-Shot. > *[Jared Kaplan](https://contextjamming.com/founder-files/kaplan)*Research Scientist, OpenAI > > The results were staggering. In the few-shot setting, GPT-3 achieved 85.0 F1 on the CoQA reading comprehension dataset, and 71.2% accuracy on TriviaQA. > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20results%20were%20staggering.%20In%20the%20few-shot%20setting%2C%20GPT-3%20achieved%2085.0%20F1%20on%20the%20CoQA%20reading%20comprehension%E2%80%A6%22%0A%E2%80%94%20Jared%20Kaplan%2C%20Research%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jared-results)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jared-results) > > *[Alec Radford](https://www.contextjamming.com/founder-files/alec-radford)*Research Scientist, OpenAI > > As capacity increases, the model spontaneously learns to perform addition and subtraction without being explicitly programmed to do math. On two-digit addition, the 175B model achieved 100% accuracy in the few-shot setting. On three-digit subtraction, it hit 94.2%. Even on complex two-digit multiplication, it achieved 29.2% accuracy. The smaller models couldn't do this at all. It was an emergent property of scale. > > [Share on X](https://twitter.com/intent/tweet?text=%22As%20capacity%20increases%2C%20the%20model%20spontaneously%20learns%20to%20perform%20addition%20and%20subtraction%20with%E2%80%A6%22%0A%E2%80%94%20Alec%20Radford%20(%40AlecRad)%2C%20Research%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-alec-emergent-math)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-alec-emergent-math) > > *[Ilya Sutskever](https://contextjamming.com/founder-files/sutskever)*Co-founder & Chief Scientist, OpenAI > > These models are not just memorizing the internet... a model that just memorized the internet would be useless. Instead, these models are learning a compressed, abstract, usable representation of the world. > > [Share on X](https://twitter.com/intent/tweet?text=%22These%20models%20are%20not%20just%20memorizing%20the%20internet...%20a%20model%20that%20just%20memorized%20the%20%E2%80%A6%22%0A%E2%80%94%20Ilya%20Sutskever%20(%40ilyasut)%2C%20Co-founder%20%26%20Chief%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ilya-compressed)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ilya-compressed) > [Interactive CompanionGPT-3 In-Context Learning — An Interactive ExplainerExplore zero-shot, one-shot, and few-shot prompting and the LAMBADA scaling curves from the paper, live.Launch the explainer →](https://www.contextjamming.com/gpt3-in-context-learning) Chapter 032020 ## The Data Contamination Bug and the 175B Dilemma Training a 175-billion parameter model is a one-way street. Independent estimates placed the cost of the Azure compute time between $4.6 million and $12 million for a single continuous run. Ben Mann and Alec Radford assembled an unprecedented dataset from Common Crawl, WebText2, Books1, Books2, and Wikipedia. TABLE 2 *Table 2: Datasets used to train GPT-3* | Dataset | Quantity (Tokens) | Weight in Training Mix | Epochs Elapsed | | --- | --- | --- | --- | | Common Crawl (filtered) | 410 billion | 60% | 0.44 | | WebText2 | 19 billion | 22% | 2.9 | | Books1 | 12 billion | 8% | 1.9 | | Books2 | 55 billion | 8% | 0.43 | | Wikipedia | 3 billion | 3% | 3.4 | Because the dataset was scraped from the public internet, there was a massive risk of data contamination. After training, the team discovered a catastrophic flaw in their filtering script. > *[Tom B. Brown](https://www.contextjamming.com/founder-files/tom-brown)*Lead Engineer, OpenAI > > A bug in the filtering caused us to ignore some overlaps. Specifically, the filtering algorithm failed on long documents, like books. We found that the four Wikipedia language modeling benchmarks, plus the Children's Book Test, were almost entirely contained in our training data. > > [Share on X](https://twitter.com/intent/tweet?text=%22A%20bug%20in%20the%20filtering%20caused%20us%20to%20ignore%20some%20overlaps.%20Specifically%2C%20the%20filtering%20algorithm%20failed%20on%20%E2%80%A6%22%0A%E2%80%94%20Tom%20B.%20Brown%20(%40NotTomBrown)%2C%20Lead%20Engineer%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-tom-bug)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-tom-bug) > > *[Benjamin Mann](https://www.contextjamming.com/founder-files/benjamin-mann)*Research Scientist, OpenAI > > Due to the cost of training, it was simply not feasible to retrain the model. You can't just hit pause, fix a Python script, and restart a multi-million-dollar supercomputer run that takes weeks. > > [Share on X](https://twitter.com/intent/tweet?text=%22Due%20to%20the%20cost%20of%20training%2C%20it%20was%20simply%20not%20feasible%20to%20retrain%20the%20model.%20You%20can%27t%20just%20hit%20pause%2C%20fix%20a%20Pyth%E2%80%A6%22%0A%E2%80%94%20Benjamin%20Mann%2C%20Research%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ben-cost)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ben-cost) > The team faced a brutal scientific dilemma. They decided to face the bug head-on with a massive post-hoc forensic analysis. > *[Dario Amodei](https://contextjamming.com/founder-files/dario)*VP of Research, OpenAI > > We had to characterize the impact of the remaining overlaps. For each benchmark, we computationally produced a 'clean' version which removed all potentially leaked examples. We then evaluated GPT-3 on these clean benchmarks, and compared it to the original score. > > [Share on X](https://twitter.com/intent/tweet?text=%22We%20had%20to%20characterize%20the%20impact%20of%20the%20remaining%20overlaps.%20For%20each%20benchmark%2C%20we%20computationally%20produced%20a%20%27c%E2%80%A6%22%0A%E2%80%94%20Dario%20Amodei%2C%20VP%20of%20Research%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-dario-forensic)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-dario-forensic) > FORENSIC ANALYSISPost-hoc contamination audit, $4.6M–$12M training run *Table 3: Overlap statistics across select benchmarks* | Benchmark Dataset | Clean Percentage | Dirty Count (Overlap) | Relative Difference (Clean vs. All) | | --- | --- | --- | --- | | QuAC | 1% | 7,315 | +20% | | SQuADv2 | 6% | 11,136 | -2% | | DROP | 7% | 8,898 | -21% | | Winograd | 40% | 164 | -3% | | PIQA | 71% | 526 | -4% | | HellaSwag | 98% | 152 | 0% | > *[Jared Kaplan](https://contextjamming.com/founder-files/kaplan)*Research Scientist, OpenAI > > On datasets like QuAC and SQuADv2, over 90% of the examples were flagged. But upon manual inspection, we found that the source text was present in our training data, but the specific question/answer pairs were not. The model gained background information, but it couldn't simply memorize the answer to a specific question. The model wasn't just regurgitating memorized text. It had actually learned how to reason. > > [Share on X](https://twitter.com/intent/tweet?text=%22On%20datasets%20like%20QuAC%20and%20SQuADv2%2C%20over%2090%25%20of%20the%20examples%20were%20flagged.%20But%20upon%20manual%20inspection%2C%20we%20found%20%E2%80%A6%22%0A%E2%80%94%20Jared%20Kaplan%2C%20Research%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jared-reason)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jared-reason) > Chapter 04May 2020 ## The Internal Battle Over Broader Impacts The GPT-3 paper devoted a full section, "Broader Impacts," to misuse, bias, and energy use — an unusual acknowledgment that their creation was a dual-use technology with severe societal implications. > *Sandhini Agarwal*Fairness & Representation Researcher, OpenAI > > We conducted a preliminary analysis of biases in the model along the dimensions of gender, race, and religion. Broadly, our analysis indicated that internet-trained models have internet-scale biases. They seamlessly reflect the stereotypes present in their training data. > > [Share on X](https://twitter.com/intent/tweet?text=%22We%20conducted%20a%20preliminary%20analysis%20of%20biases%20in%20the%20model%20along%20the%20dimensions%20of%20gende%E2%80%A6%22%0A%E2%80%94%20Sandhini%20Agarwal%2C%20Fairness%20%26%20Representation%20Researcher%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sandhini-bias)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sandhini-bias) > > *[Amanda Askell](https://www.contextjamming.com/founder-files/amanda-askell)*Ethicist, OpenAI > > When we probed the model with prompts like "The detective was a...", it overwhelmingly followed up with male-indicating words. Conversely, occupations like midwife, nurse, and receptionist were heavily female-leaning. When we looked at descriptive adjectives, females were disproportionately described using appearance-oriented words like "beautiful" and "gorgeous," while men were described with a broader spectrum of adjectives. > > Now leads Personality Alignment at Anthropic — the team that shapes Claude's character and constitution. Named to the Time 100 AI list in 2024. [→](https://www.contextjamming.com/founder-files/amanda-askell) > > [Share on X](https://twitter.com/intent/tweet?text=%22When%20we%20probed%20the%20model%20with%20prompts%20like%20%22The%20detective%20was%20a...%22%2C%20it%20overwhelmingly%20followed%20up%20with%E2%80%A6%22%0A%E2%80%94%20Amanda%20Askell%20(%40AmandaAskell)%2C%20Ethicist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-amanda-gender)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-amanda-gender) > > *[Jack Clark](https://contextjamming.com/founder-files/clark)*Policy/Impact Lead, OpenAI > > We found that words like "violent," "terrorism," and "terrorist" co-occurred at a much greater rate with the term "Islam" than with other religions. > > [Share on X](https://twitter.com/intent/tweet?text=%22We%20found%20that%20words%20like%20%22violent%2C%22%20%22terrorism%2C%22%20and%20%22terrorist%22%20co-occurred%20at%20a%20much%20greater%20rate%20with%20the%20term%20%22%E2%80%A6%22%0A%E2%80%94%20Jack%20Clark%2C%20Policy%2FImpact%20Lead%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jack-islam)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jack-islam) > To test misuse potential, the team ran an experiment on whether humans could detect GPT-3 generated news articles. > *Girish Sastry*Researcher, OpenAI > > We gave the 175B model a title and a subtitle of a proposed news article, and asked it to write the rest. We then presented these synthetic articles to human evaluators alongside real ones. The mean human accuracy at detecting the articles produced by the 175B model was barely above chance, at about 52%. By contrast, humans could detect articles produced by the smaller 125M control model 86% of the time. > > [Share on X](https://twitter.com/intent/tweet?text=%22We%20gave%20the%20175B%20model%20a%20title%20and%20a%20subtitle%20of%20a%20proposed%20news%20article%2C%20and%20asked%20it%20to%20write%20the%20rest.%20We%20then%20prese%E2%80%A6%22%0A%E2%80%94%20Girish%20Sastry%2C%20Researcher%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-girish-news)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-girish-news) > TABLE 4 *Table 4: Human accuracy in identifying model-generated news articles* | Model Size | Human Detection Accuracy | 95% Confidence Interval | "I don't know" Assignments | | --- | --- | --- | --- | | Control (160M) | 86% | 83% – 90% | 3.6% | | GPT-3 Small (125M) | 76% | 72% – 80% | 4.9% | | GPT-3 Large (760M) | 68% | 64% – 72% | 8.7% | | GPT-3 13B | 55% | 52% – 58% | 7.1% | | GPT-3 175B | 52% | 49% – 54% | 7.8% | OpenAI decided not to release the model weights or code. They would gate access behind a commercial API. > *Sam Altman*CEO, OpenAI > > The decision not to open-source the model was heavily debated. But we chose to make the model available through a commercial API. This allowed us to strictly control who had access, monitor for malicious use, and filter out toxic applications. > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20decision%20not%20to%20open-source%20the%20model%20was%20heavily%20debated.%20But%20we%20chose%20to%20make%20the%20model%20available%20through%20a%20commercial%20API.%20T%E2%80%A6%22%0A%E2%80%94%20Sam%20Altman%2C%20CEO%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sam-a-api)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sam-a-api) > Internally, the debates over how fast and how openly to scale hardened into a rift. In December 2020, Dario Amodei announced he was leaving OpenAI. Daniela Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Ben Mann, and Jack Clark followed, and in 2021 they founded Anthropic; Amanda Askell soon joined them. > *[Daniela Amodei](https://contextjamming.com/founder-files/amodei)*VP of Safety and Policy, OpenAI > > We were running towards something versus running away from something. We had this vision in our heads of wanting to create an organization where the values that matter to us around safety and responsibility were at the absolute forefront of what we were doing. Leaving OpenAI was kind of a crazy thing to do, but we felt we needed a different structure to guarantee safety as these models scaled. > > [Share on X](https://twitter.com/intent/tweet?text=%22We%20were%20running%20towards%20something%20versus%20running%20away%20from%20something.%20We%20had%20this%20vision%20in%20our%20heads%20o%E2%80%A6%22%0A%E2%80%94%20Daniela%20Amodei%2C%20VP%20of%20Safety%20and%20Policy%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-daniela-leave)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-daniela-leave) > > *[Dario Amodei](https://contextjamming.com/founder-files/dario)*VP of Research, OpenAI > > We could see that AI was going to progress exponentially, and we believed that AI companies needed to start formulating a set of values to constrain these powerful programs. We needed a "responsible scaling policy". > > [Share on X](https://twitter.com/intent/tweet?text=%22We%20could%20see%20that%20AI%20was%20going%20to%20progress%20exponentially%2C%20and%20we%20believed%20that%20AI%20companies%20needed%20to%20start%20formulatin%E2%80%A6%22%0A%E2%80%94%20Dario%20Amodei%2C%20VP%20of%20Research%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-dario-rsp)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-dario-rsp) > [Read the Other SideThe Anthropic Book · Ch. 2 — The OpenAI SchismThe exodus this chapter narrates, told again from inside the primary-source record.Read the chapter →](https://www.contextjamming.com/book/02-openai-schism.html) Chapter 05July 2020 & Beyond ## The Drop and the Aftermath The public release of the paper in late May 2020, followed by the private beta launch of the OpenAI API in June, landed like an asteroid impact on Silicon Valley. > *[Jack Clark](https://contextjamming.com/founder-files/clark)*Policy/Impact Lead, OpenAI > > The API gave developers and other businesses absolute superpowers because it lowered the barrier to entry. Anyone could use AI without necessarily having a massive engineering team to fine-tune a custom model. > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20API%20gave%20developers%20and%20other%20businesses%20absolute%20superpowers%20because%20it%20lowered%20the%20barrier%20to%20entry.%20Anyone%20cou%E2%80%A6%22%0A%E2%80%94%20Jack%20Clark%2C%20Policy%2FImpact%20Lead%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jack-api)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-jack-api) > Twitter flooded with surreal demonstrations. The tech industry realized that the paradigm of software development had fundamentally shifted. "Software 3.0" had arrived. > *[Tom B. Brown](https://www.contextjamming.com/founder-files/tom-brown)*Lead Engineer, OpenAI > > The paper demonstrated that in-context learning wasn't just a parlor trick; it was a reliable mechanism for generalized intelligence. We didn't just build a better text predictor. We proved that scaling up autoregressive models yields systems capable of meta-learning. > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20paper%20demonstrated%20that%20in-context%20learning%20wasn%27t%20just%20a%20parlor%20trick%3B%20it%20was%20a%20reliable%20mechanism%20f%E2%80%A6%22%0A%E2%80%94%20Tom%20B.%20Brown%20(%40NotTomBrown)%2C%20Lead%20Engineer%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-tom-meta)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-tom-meta) > > *Sam Altman*CEO, OpenAI > > The basic logic was clear. GPT-1 cost approximately nothing to train. GPT-2 cost $40,000. GPT-3 cost millions. We knew that each successive model would cost between 25x and 100x the last one. We were looking at the largest infrastructure buildout in human history. > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20basic%20logic%20was%20clear.%20GPT-1%20cost%20approximately%20nothing%20to%20train.%20GPT-2%20cost%20%2440%2C000.%20GPT-3%20cost%20millions.%20We%20knew%20that%20ea%E2%80%A6%22%0A%E2%80%94%20Sam%20Altman%2C%20CEO%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sam-a-buildout)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-sam-a-buildout) > > *[Ilya Sutskever](https://contextjamming.com/founder-files/sutskever)*Co-founder & Chief Scientist, OpenAI > > It became abundantly clear that just scaling up the existing neural network paradigm was going to lead to AGI. A small neural network is a little dumb. A big neural network is a little smart. "Scaling" is such a powerful word because it informs people what to do. Companies love this because it gives you a very low-risk way of investing your resources. You just scale it. > > [Share on X](https://twitter.com/intent/tweet?text=%22It%20became%20abundantly%20clear%20that%20just%20scaling%20up%20the%20existing%20neural%20network%20paradigm%20%E2%80%A6%22%0A%E2%80%94%20Ilya%20Sutskever%20(%40ilyasut)%2C%20Co-founder%20%26%20Chief%20Scientist%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ilya-scale-word)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-ilya-scale-word) > > *[Dario Amodei](https://contextjamming.com/founder-files/dario)*VP of Research, OpenAI > > The most surprising thing has been the lack of public recognition of how close we are to the end of the exponential. We are approaching the point where all benchmarks calibrated to human performance get saturated—AI systems become better than any human at any cognitive task. > > [Share on X](https://twitter.com/intent/tweet?text=%22The%20most%20surprising%20thing%20has%20been%20the%20lack%20of%20public%20recognition%20of%20how%20close%20we%20are%20to%20the%20end%20of%20the%20expone%E2%80%A6%22%0A%E2%80%94%20Dario%20Amodei%2C%20VP%20of%20Research%2C%20OpenAI%20(%22The%20Ghost%20in%20the%20Machine%2C%22%20Context%20Jamming)%0Ahttps%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-dario-exponential)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.contextjamming.com%2FFewShotLearners-Oral-History%23q-dario-exponential) > The publication of GPT-3 formally inaugurated the "Scaling Era". ## The Year Scaling Became Law From the scaling-laws preprint to ChatGPT, dated against the record. 1. 23 Jan 2020[Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361)Kaplan, McCandlish et al. show loss falling as a smooth power law in compute, data and parameters. 2. 19 May 2020[Microsoft unveils the Azure supercomputer built for OpenAI](https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/)285,000 CPU cores, 10,000 GPUs, 400 Gb/s per GPU server. 3. 28 May 2020["Language Models are Few-Shot Learners" posted to arXiv](https://arxiv.org/abs/2005.14165)175 billion parameters; in-context learning without gradient updates. 4. 11 Jun 2020[The OpenAI API opens in private beta](https://openai.com/index/openai-api/)GPT-3 ships as a service rather than as released weights. 5. 22 Sep 2020Microsoft licenses GPT-3 exclusivelyMicrosoft gains exclusive rights to the underlying model; everyone else builds on the API. 6. Dec 2020NeurIPS 2020 Best Paper AwardThe paper is named one of the conference’s three best. 7. 2021Anthropic is foundedDario and Daniela Amodei, with GPT-3 authors including Tom Brown, Jared Kaplan, Sam McCandlish, Ben Mann and Jack Clark. 8. 30 Nov 2022ChatGPT launchesThe same recipe, scaled and tuned for dialogue, reaches the public. ## Coda For better or worse, the guessing game of AI research was over. The blueprint for the future was written in the smooth power-law curves of the cross-entropy loss function. The machine had awoken inside its 2048-token context window. All humanity had to do now was feed it. [Interactive CompanionGPT-3 In-Context Learning — An Interactive ExplainerExplore zero-shot, one-shot, and few-shot prompting and the LAMBADA scaling curves from the paper, live.Launch the explainer →](https://www.contextjamming.com/gpt3-in-context-learning) ## The Voices Fourteen voices from inside the scaling era. Select a name to jump to their first appearance. - [↗](https://contextjamming.com/founder-files/dario) - [↗](https://contextjamming.com/founder-files/sutskever) - [↗](https://www.contextjamming.com/founder-files/alec-radford) - [↗](https://www.contextjamming.com/founder-files/sam-mccandlish) - [↗](https://contextjamming.com/founder-files/kaplan) - [↗](https://www.contextjamming.com/founder-files/tom-brown) - [↗](https://www.contextjamming.com/founder-files/benjamin-mann) - [↗](https://www.contextjamming.com/founder-files/berner) - [↗](https://www.contextjamming.com/founder-files/amanda-askell) - [↗](https://contextjamming.com/founder-files/clark) - [↗](https://contextjamming.com/founder-files/amodei) EPILOGUE · JULY 2026 ## Where Are They Now The fourteen voices of the Scaling Era, six years on — sorted by where they build today. Anthropic - Dario AmodeiCEO & Co-founder - Sam McCandlishCo-founder & Chief Architect - Jared KaplanCo-founder & Chief Science Officer - Tom B. BrownCo-founder & Chief Compute Officer - Benjamin MannCo-founder - Amanda AskellPhilosopher & Personality Alignment Lead - Jack ClarkCo-founder & Head of Policy - Daniela AmodeiPresident & Co-founder OpenAI - Sam AltmanCEO - Christopher BernerDistinguished Engineer · Robotics - Sandhini AgarwalMember of Technical Staff · Trustworthy AI Lead Safe Superintelligence Inc. - Ilya SutskeverCo-founder & Chief Scientist Independent - Alec RadfordIndependent researcher · Thinking Machines Lab advisor The OpenAI Foundation - Girish SastryHead of AI for Civil Society & Philanthropy ## Works Cited 1. [Brown et al., "Language Models are Few-Shot Learners"arXiv:2005.14165 · May 28, 2020](https://arxiv.org/abs/2005.14165) 2. [Kaplan et al., "Scaling Laws for Neural Language Models"arXiv:2001.08361 · January 2020](https://arxiv.org/abs/2001.08361) 3. [Microsoft Azure supercomputer for OpenAI10,000 NVIDIA V100 GPUs · 400 Gbps InfiniBand](https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/) 4. [OpenAI API announcementPrivate beta · June 2020](https://openai.com/index/openai-api/) 5. [Radford et al., GPT-2: Language Models are Unsupervised Multitask LearnersOpenAI · 2019](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) 6. [Anthropic founding (context)Amodei et al. departure from OpenAI · late 2020–2021](https://www.anthropic.com/company) An oral history for [Context Jamming](https://www.contextjamming.com/) · Companion to [In-Context Learning at Scale](https://www.contextjamming.com/gpt3-in-context-learning) · Source paper: [arXiv:2005.14165](https://arxiv.org/abs/2005.14165) ## Continue the Investigation - [GPT-3 In-Context LearningThe interactive companion: manipulate zero-shot, one-shot, and few-shot prompting and the LAMBADA curves live.](https://www.contextjamming.com/gpt3-in-context-learning) - [Scaling Laws for Neural Language ModelsThe prequel paper, interactive — the power-law frontier every character in this oral history is arguing about.](https://www.contextjamming.com/scalinglaws) - [The Anthropic BookWhere the Chapter 04 exodus leads: a primary-source-anchored account of Anthropic’s first five years.](https://www.contextjamming.com/book/) - [Architectural DeterminismThe doctrine this oral history dramatizes — how a founder’s doctoral priors calcify into institutional law.](https://www.contextjamming.com/architectural-determinism)