Reference
AI safety glossary
Plain-English definitions of the terms behind Dario Amodei's essays and Anthropic's research.
Short definitions of terms that appear across this site.
- Agentic AI
- AI systems that take multi-step actions toward a goal, such as writing and running code, rather than only answering a single prompt.
- AI Safety Level (ASL)
- A tier in Anthropic's Responsible Scaling Policy that pairs a level of model capability with required safeguards.
- Alignment
- The effort to make AI systems pursue the goals and values their developers and users intend.
- Compute
- The processing power, typically measured in operations, used to train or run an AI model.
- Constitutional AI
- An Anthropic training method in which a model critiques and revises its outputs against written principles.
- Context window
- The amount of text a model can consider at once when generating a response.
- Export controls
- Government restrictions on exporting certain technologies, such as advanced AI chips, to specific countries.
- Feature
- In interpretability, a pattern of neural activity that corresponds to a human-understandable concept.
- Frontier model
- One of the most capable AI models available at a given time.
- Interpretability
- Research aimed at understanding the internal workings of neural networks.
- Jailbreak
- A prompt designed to get a model to ignore its safety training.
- Long-Term Benefit Trust
- A group of independent trustees at Anthropic designed to help balance public benefit with commercial incentives.
- Powerful AI
- Amodei's term for AI systems exceeding top human experts across most fields, often summarised as a “country of geniuses in a datacenter”.
- Public benefit corporation
- A for-profit company legally permitted to pursue a stated public benefit alongside profit.
- Red teaming
- Deliberately probing a system for weaknesses or harmful behaviours before release.
- Reward hacking
- When an AI system finds unintended ways to score well on its objective without doing what designers wanted.
- RLAIF
- Reinforcement learning from AI feedback; AI-generated preferences replace some human labels.
- RLHF
- Reinforcement learning from human feedback; human preferences guide a model's training.
- Scalable oversight
- Techniques for supervising AI systems that may be more capable than their human overseers.
- Scaling laws
- Empirical relationships showing how model performance improves with size, data and compute.
- Superposition
- In interpretability, the way networks can represent more concepts than they have neurons by overlapping them.