The Science of Artificial Intelligence
How neural networks, transformers and scaling laws produce modern AI.
Modern AI systems are transformers trained by next-token prediction. GPT-3 used 175 billion parameters. Training roughly costs 6 FLOPs per parameter per token. Kaplan et al. showed loss falls as a power law with compute, parameters and data.

Three numbers
- 175 billion — number of parameters in the largest GPT-3 model. Source: OpenAI, “Language Models are Few-Shot Learners” (Brown et al., 2020).
- ~6 FLOPs — approximate floating-point operations per parameter per training token (forward + backward pass heuristic). Source: Kaplan et al. / OpenAI scaling analyses and GPT-3 compute tables.
- Power law — cross-entropy loss scales as a power law with model size, dataset size and training compute. Source: Kaplan et al., “Scaling Laws for Neural Language Models” (2020).
What the data shows
A common claim is that artificial intelligence works by some form of genuine understanding or magical reasoning. The measured reality is narrower: modern large language models are transformer neural networks trained to predict the next token in a sequence. Given preceding text, the model outputs a probability distribution over the next token; training minimizes the error of those predictions across enormous corpora. No explicit rules or symbolic knowledge are programmed in. Capabilities emerge from statistical patterns in the data.
The architecture that made this practical at scale is the transformer, introduced in the 2017 paper “Attention Is All You Need” (Vaswani et al.). Instead of processing sequences step by step (as earlier recurrent networks did), the transformer uses self-attention: every token can directly attend to every other token in the input via learned query, key and value projections. This removes the sequential bottleneck and allows full parallelization on GPUs. Multiple attention heads and stacked layers let the model capture different types of relationships. GPT-3, described in the 2020 OpenAI paper, scaled this design to 175 billion parameters and was trained on roughly 300 billion tokens. Its compute requirement was estimated at about 3.14 × 10²³ FLOPs using the rule of thumb of approximately six floating-point operations per parameter per token.
Performance is governed by empirical scaling laws. Kaplan et al. (2020) showed that cross-entropy loss falls as a power law with the number of parameters, the number of training tokens and the total training compute, with trends spanning many orders of magnitude. Later work (Chinchilla / Hoffmann et al., 2022) refined the optimal split between model size and data for a given compute budget. The core finding remains: more compute, more parameters and more data reliably improve next-token prediction, and many downstream abilities improve with it. The science of contemporary AI is therefore the science of scaled statistical prediction under the transformer architecture and these power-law relationships.
| Element | What the data shows |
|---|---|
| Core task | Next-token prediction |
| Architecture | Transformer with self-attention (Vaswani et al. 2017) |
| GPT-3 scale | 175 billion parameters |
| Training cost heuristic | ~6 FLOPs per parameter per token |
| Performance scaling | Power-law in parameters, data and compute (Kaplan et al. 2020) |
Parameter count and training details from Brown et al. / OpenAI GPT-3 paper. Compute heuristic from Kaplan-style analyses and GPT-3 tables. Scaling laws from Kaplan et al. 2020. Architecture from Vaswani et al. 2017.
How it was measured
Transformer details come from the original 2017 paper. GPT-3’s 175 billion parameter count, training token volume and FLOPs estimate are taken from the OpenAI 2020 paper and its supplementary tables. The approximately 6 FLOPs-per-parameter-per-token rule is the standard forward-plus-backward heuristic used in those analyses (two operations in the forward pass, roughly three times that for the backward pass). Scaling laws are the empirical power-law fits reported by Kaplan et al. across wide ranges of model size and compute. Later refinements (Chinchilla) adjust the optimal data-versus-parameter allocation but do not overturn the basic power-law observation.
How we built this
- Dataset: Vaswani et al. 2017 (Attention Is All You Need). Brown et al. / OpenAI 2020 (GPT-3). Kaplan et al. 2020 (scaling laws).
- Scope: Core mechanism of modern large language models: transformer architecture, next-token training, and empirical scaling.
- Figures: 175 billion parameters. ~6 FLOPs per parameter per token. Power-law scaling of loss.






