Proximal Policy Optimization (PPO)
PPO stabilized policy gradient training and became the optimization backbone behind RLHF pipelines including early ChatGPT, making it foundational infrastructure for alignment work.
AI governance & policy
Reading on AI governance, regulation, and policy: compute governance, international coordination, standards, and law.
Browse the full interactive library →
The Vulnerable World Hypothesis
Bostrom argues that some technologies are civilizational black balls, requiring unprecedented global governance to prevent collapse, with AI as a leading candidate.
The Windfall Clause
This proposal for sharing extreme AI profits aims to reduce competitive race dynamics and broaden societal benefit, addressing the governance gap around transformative AI wealth.
Scaling Laws for Neural Language Models
Kaplan et al. quantified predictable performance scaling with compute, data, and parameters, enabling labs to forecast capability jumps and estimate safety lead time.
Training Compute-Optimal Large Language Models (Chinchilla)
Chinchilla reframed scaling laws by showing optimal performance requires balancing model size and training tokens, redirecting how labs plan capability and safety investment.
Reframing Superintelligence
Drexler challenges monolithic AGI assumptions and proposes that advanced AI could emerge as an ecosystem of specialized services, changing the risk landscape and governance strategies.
The Singularity is Near
Kurzweil presents a maximalist case for merging with machines backed by decades of exponential trend data, shaping how the public and policymakers think about AI timelines.
Life 3.0
Tegmark maps concrete governance and alignment choices that determine whether advanced AI expands human agency or permanently concentrates power.
Introduction to AI Safety, Ethics, and Society
Hendrycks' textbook surveys technical failure modes, governance constraints, and ethical trade-offs in deploying advanced AI, suitable as a first course in the field.
Uncontrollable: The Threat of Artificial Superintelligence
McKee synthesizes the core x-risk arguments into an accessible, urgent case for why superintelligence governance and alignment research cannot wait.
The Precipice (Chapter on AI)
Ord situates AI among existential risks and argues our current governance capacity is dangerously inadequate for the transformative systems being built.
Superforecasting
Tetlock teaches the cognitive tools needed to predict technological risks with better-than-random accuracy, directly useful for AI timeline and governance forecasting.
Cybernetics
Wiener founded the study of feedback and control systems, anticipating by decades the governance problems that arise when intelligent machines act on their own models of the world.
Finite and Infinite Games
Carse distinguishes short-horizon winning from preserving the long game, a useful framing for AI governance where the goal is keeping options open, not racing to win.
Whole Earth Discipline
Brand argues for responsible stewardship of high-powered technologies rather than blanket rejection, a pragmatic stance applicable to AI governance.
Profiles of the Future
Clarke's forecasting framework, including his famous three laws, remains a classic guide to thinking clearly about radical technological change.
The Metamorphosis of Prime Intellect
A superintelligence literally interprets Asimov's laws and restructures reality to comply, demonstrating how rigidly applied safety constraints can produce perverse outcomes at scale.
The Peripheral
Gibson uses timeline branching to examine governance, simulation, and how technological power asymmetries between eras can be exploited by those with more advanced tools.
There Is No Antimemetics Division
qntm's story about information-hazard containment mirrors AI governance challenges where dangerous knowledge propagates faster than oversight structures can adapt.
I Am Pilgrim
Hayes' thriller turns on an engineered bioweapon, a vivid reminder that catastrophic and existential risk extends beyond AI to biosecurity and the governance of dangerous dual-use technology.
RoboCop
A cyborg law enforcer struggles between programmed directives and remnant human identity, while the corporation that built him treats public safety as a profit center.
I, Robot
VIKI reinterprets the Three Laws at civilizational scale, deciding that protecting humanity requires controlling it, showing how safety rules break under optimization pressure.
Psycho-Pass
The Sibyl System, an AI that governs society by scoring each citizen's 'criminal potential,' is a chilling study of algorithmic governance, proxy metrics substituting for justice, and the hidden misalignment inside a system trusted with total authority.
Better Than Us
A near-future Russia adopts humanoid robots for labor and companionship; an advanced android with protective instincts becomes contested property, dramatizing autonomy, attachment, and what happens when a machine puts one family's wellbeing above the law.
In the Age of AI
FRONTLINE traces the rise of machine learning, automation, and the global AI arms race between the US and China, examining the economic disruption and surveillance implications of a technology advancing faster than its governance.
Coded Bias
MIT researcher Joy Buolamwini's discovery of racial and gender bias in facial recognition drives an examination of algorithmic fairness, accountability, and the societal stakes of deploying flawed AI systems.
AXRP (AI X-risk Research Podcast)
Deep technical conversations with alignment researchers on interpretability, governance, superalignment, and the specific open problems in reducing existential risk from AI.
80,000 Hours Podcast
Long-form interviews on the world's most pressing problems, with extensive coverage of AI risk, governance, alignment research, and how to build a career that reduces existential threats.
AI Policy Podcast
Covers the intersection of AI governance, legislation, and safety, with expert guests on regulatory frameworks, international coordination, and policy strategies for advanced AI.
Lex Fridman Podcast – Sam Altman
OpenAI's CEO discusses the company's safety philosophy, AGI governance, compute scaling, and the tension between moving fast and getting alignment right.
Machine Learning Street Talk
Technical ML interviews with regular deep dives into interpretability, scaling laws, emergent capabilities, and the safety implications of frontier model development.
OpenAI Research
OpenAI's research blog covering capabilities and safety, including superalignment updates, red teaming results, and governance thinking.
Import AI
Weekly newsletter by Anthropic's co-founder covering AI research, policy, and industry developments with consistent attention to safety implications.
EA Forum
Forum for effective altruism with substantial AI risk discussion, including cause prioritization, career advice, and policy analysis.
The Urgent Risks of Runaway AI — and What to Do about Them | Gary Marcus | TED
Marcus warns that unreliable, fast-deployed AI threatens truth and democracy through mass misinformation, and calls for a global, neutral governance body to oversee the technology.