Home AI Technology Gaming Future Economy Geography Automobile Science

Explained: The Science of Artificial Intelligence

Macro Discovery
On: October 31, 2026 7:09 AM
Follow Us:
Explained: The Science of Artificial Intelligence | Macro Discovery
Explained · Artificial Intelligence · Machine Learning

The Science of Artificial Intelligence

How neural networks, transformers and scaling laws produce modern AI.

Modern AI systems are transformers trained by next-token prediction. GPT-3 used 175 billion parameters. Training roughly costs 6 FLOPs per parameter per token. Kaplan et al. showed loss falls as a power law with compute, parameters and data.

175B
GPT-3 parameters
~6
FLOPs / param / token
Power law
Scaling of loss
Science of Artificial Intelligence
Science of Artificial Intelligence

Three numbers

  • 175 billion — number of parameters in the largest GPT-3 model. Source: OpenAI, “Language Models are Few-Shot Learners” (Brown et al., 2020).
  • ~6 FLOPs — approximate floating-point operations per parameter per training token (forward + backward pass heuristic). Source: Kaplan et al. / OpenAI scaling analyses and GPT-3 compute tables.
  • Power law — cross-entropy loss scales as a power law with model size, dataset size and training compute. Source: Kaplan et al., “Scaling Laws for Neural Language Models” (2020).

What the data shows

A common claim is that artificial intelligence works by some form of genuine understanding or magical reasoning. The measured reality is narrower: modern large language models are transformer neural networks trained to predict the next token in a sequence. Given preceding text, the model outputs a probability distribution over the next token; training minimizes the error of those predictions across enormous corpora. No explicit rules or symbolic knowledge are programmed in. Capabilities emerge from statistical patterns in the data.

The architecture that made this practical at scale is the transformer, introduced in the 2017 paper “Attention Is All You Need” (Vaswani et al.). Instead of processing sequences step by step (as earlier recurrent networks did), the transformer uses self-attention: every token can directly attend to every other token in the input via learned query, key and value projections. This removes the sequential bottleneck and allows full parallelization on GPUs. Multiple attention heads and stacked layers let the model capture different types of relationships. GPT-3, described in the 2020 OpenAI paper, scaled this design to 175 billion parameters and was trained on roughly 300 billion tokens. Its compute requirement was estimated at about 3.14 × 10²³ FLOPs using the rule of thumb of approximately six floating-point operations per parameter per token.

Performance is governed by empirical scaling laws. Kaplan et al. (2020) showed that cross-entropy loss falls as a power law with the number of parameters, the number of training tokens and the total training compute, with trends spanning many orders of magnitude. Later work (Chinchilla / Hoffmann et al., 2022) refined the optimal split between model size and data for a given compute budget. The core finding remains: more compute, more parameters and more data reliably improve next-token prediction, and many downstream abilities improve with it. The science of contemporary AI is therefore the science of scaled statistical prediction under the transformer architecture and these power-law relationships.

ElementWhat the data shows
Core taskNext-token prediction
ArchitectureTransformer with self-attention (Vaswani et al. 2017)
GPT-3 scale175 billion parameters
Training cost heuristic~6 FLOPs per parameter per token
Performance scalingPower-law in parameters, data and compute (Kaplan et al. 2020)

Parameter count and training details from Brown et al. / OpenAI GPT-3 paper. Compute heuristic from Kaplan-style analyses and GPT-3 tables. Scaling laws from Kaplan et al. 2020. Architecture from Vaswani et al. 2017.

How it was measured

Transformer details come from the original 2017 paper. GPT-3’s 175 billion parameter count, training token volume and FLOPs estimate are taken from the OpenAI 2020 paper and its supplementary tables. The approximately 6 FLOPs-per-parameter-per-token rule is the standard forward-plus-backward heuristic used in those analyses (two operations in the forward pass, roughly three times that for the backward pass). Scaling laws are the empirical power-law fits reported by Kaplan et al. across wide ranges of model size and compute. Later refinements (Chinchilla) adjust the optimal data-versus-parameter allocation but do not overturn the basic power-law observation.

How we built this

  • Dataset: Vaswani et al. 2017 (Attention Is All You Need). Brown et al. / OpenAI 2020 (GPT-3). Kaplan et al. 2020 (scaling laws).
  • Scope: Core mechanism of modern large language models: transformer architecture, next-token training, and empirical scaling.
  • Figures: 175 billion parameters. ~6 FLOPs per parameter per token. Power-law scaling of loss.

Common questions

How do modern AI language models work?
They are transformer neural networks trained to predict the next token in a sequence. Self-attention lets the model weigh relationships between all tokens at once. Training uses gradient descent to minimize prediction error on large text corpora.
How many parameters did GPT-3 have?
175 billion parameters, as reported in the 2020 OpenAI paper “Language Models are Few-Shot Learners.” It was trained on roughly 300 billion tokens.
What are scaling laws in AI?
Empirical relationships, first quantified by Kaplan et al. (2020), showing that cross-entropy loss falls as a power law with model size, dataset size and training compute across many orders of magnitude.

Same topic cluster

Macro Discovery

Sukh Dhaliwal

Sukh Dhaliwal is the founder of Macro Discovery, an independent digital publication covering AI, technology, science, future trends, and global innovation through visual storytelling and data-driven analysis.

Join WhatsApp

Join Now

Join Facebook

Follow us

Leave a Comment