Nick Bostrom’s Home Page (original) (raw)

The most recent addition is “AI Creation and the Cosmic Host”.

Second print run of Deep Utopia, which came out a few months ago, is available. (Update: Gold Medal Winner, Living Now Book Awards 2024; Nonfiction: Cross-Genre WINNER, American Book Fest; Fiction: Intrigue WINNER, PenCraft Book Awards). 15 October: Audiobook just released!

The world is quickening, and philosophers are statues. (I take pride in having at least winked once or twice.)

For more, e.g. New Yorker profile (old), Bio, CV, Contact, Press images.

Recent additions

AI Creation and the Cosmic Host, working paper
Propositions Concerning Digital Minds and Society, w/ Carl Shulman, in The Cambridge Journal of Law, Politics, and Art
Base Camp for Mt. Ethics, working paper
Sharing the World with Digital Minds, w/ Carl Shulman, in edited volume (Oxford University Press, 2021)
The Vulnerable World Hypothesis, in Global Policy. Also German book (Suhrkamp, 2020); adaptation in Aeon
Strategic Implications of Openness in AI Development, in Global Policy
Hail Mary, Value Porosity, and Utility Diversification, working paper

Ethics & Policy

Propositions Concerning Digital Minds and Society

AIs with moral status and political rights? We'll need a modus vivendi, and it’s becoming urgent to figure out the parameters for that. This paper makes a load of specific claims that begin to stake out a position.

AI Creation and the Cosmic Host

There may well exist a normative structure, based on the preferences or concordats of a cosmic host, and which has high relevance to the development of AI.

Humans are relatively expensive but absolutely cheap.

Strategic Implications of Openness in AI Development

An analysis of the global desirability of different forms of openness (including source code, science, data, safety techniques, capabilities, and goals).

The Reversal Test: Eliminating Status Quo Bias in Applied Ethics

We present a heuristic for correcting for one kind of bias (status quo bias), which we suggest affects many of our judgments about the consequences of modifying human nature. We apply this heuristic to the case of cognitive enhancements, and argue that the consequentialist case for this is much stronger than commonly recognized.

The Fable of the Dragon-Tyrant

Recounts the Tale of a most vicious Dragon that ate thousands of people every day, and of the actions that the King, the People, and an assembly of Dragonologists took with respect thereto.

Astronomical Waste: The Opportunity Cost of Delayed Technological Development

Suns are illuminating and heating empty rooms, unused energy is being flushed down black holes, and our great common endowment of negentropy is being irreversibly degraded into entropy on a cosmic scale. These are resources that an advanced civilization could have used to create value-structures, such as sentient beings living worthwhile lives…

Infinite Ethics

Cosmology shows that we might well be living in an infinite universe that contains infinitely many happy and sad people. Given some assumptions, aggregative ethics implies that such a world contains an infinite amount of positive value and an infinite amount of negative value. But you can presumably do only a finite amount of good or bad. Since an infinite cardinal quantity is unchanged by the addition or subtraction of a finite quantity, it looks as though you can't change the value of the world. Aggregative consequentialism (and many other important ethical theories) are threatened by total paralysis. We explore a variety of potential cures, and discover that none works perfectly and all have serious side-effects. Is aggregative ethics doomed?

The Unilateralist's Curse: The Case for a Principle of Conformity

In cases where several altruistic agents each have an opportunity to undertake some initiative, a phenomenon arises that is analogous to the winner's curse in auction theory. To combat this problem, we propose a principle of conformity. It has applications in technology policy and many other areas.

Public Policy and Superintelligent AI: A Vector Field Approach

What properties should we want a proposal for an AI governance pathway to have?

Dignity and Enhancement

Does human enhancement threaten our dignity as some have asserted? Or could our dignity perhaps be technologically enhanced? After disentangling several different concepts of dignity, this essay focuses on the idea of dignity as a quality (a kind of excellence admitting of degrees). The interactions between enhancement and dignity as a quality are complex and link into fundamental issues in ethics and value theory.

In Defense of Posthuman Dignity

Brief paper, critiques a host of bioconservative pundits who believe that enhancing human capacities and extending human healthspan would undermine our dignity.

Human Enhancement

Original essays by various prominent moral philosophers on the ethics of human enhancement.

Enhancement Ethics: The State of the Debate

The introductory chapter from the book: 1–22

Human Genetic Enhancements: A Transhumanist Perspective

A transhumanist ethical framework for public policy regarding genetic enhancements, particularly human germ-line genetic engineering

Ethical Issues in Human Enhancement

Anthology chapter on the ethics of human enhancement

The Ethics of Artificial Intelligence

Overview of ethical issues raised by the possibility of creating intelligent machines. Questions relate both to ensuring such machines do not harm humans and to the moral status of the machines themselves.

Ethical Issues In Advanced Artificial Intelligence

Some cursory notes; not very in-depth.

Smart Policy: Cognitive Enhancement and the Public Interest

Short article summarizing some of the key issues and offering specific recommendations, illustrating the opportunity and need for "smart policy": the integration into public policy of a broad-spectrum of approaches aimed at protecting and enhancing cognitive capacities and epistemic performance of individuals and institutions.

Base Camp for Mt. Ethics

New theoretical ideas for a big expedition.

Transhumanism

Why I Want to be a Posthuman When I Grow Up

After some definitions and conceptual clarification, I argue for two theses. First, some posthuman modes of being would be extremely worthwhile. Second, it could be good for human beings to become posthuman.

Letter from Utopia

The good life: just how good could it be? A vision of the future from the future.

The Transhumanist FAQ

The revised version 2.1. The document represents an effort to develop a broadly based consensus articulation of the basics of responsible transhumanism. Some one hundred people collaborated with me in creating this text. Feels like from another era.

Transhumanist Values

Wonderful ways of being may be located in the "posthuman realm", but we can't reach them. If we enhance ourselves using technology, however, we can go out there and realize these values. This paper sketches a transhumanist axiology.

A History of Transhumanist Thought

The human desire to acquire new capacities, to extend life and overcome obstacles to happiness is as ancient as the species itself. But transhumanism has emerged gradually as a distinctive outlook, with no one person being responsible for its present shape. Here's one account of how it happened.

Decision Theory

Pascal's Mugging

Finite version of Pascal's Wager.

The Meta-Newcomb Problem

A self-undermining variant of the Newcomb problem.

Philosophy of Mind

Quantity of Experience: Brain-Duplication and Degrees of Consciousness

If two brains are in identical states, are there two numerically distinct phenomenal experiences or only one? Two, I argue. But what happens in intermediary cases? This paper looks in detail at this question and suggests that there can be a fractional (non-integer) number of qualitatively identical experiences. This has implications for what it is to implement a computation and for Chalmer's Fading Qualia thought experiment.

Risk & The Future

The Vulnerable World Hypothesis

Is there a level of technology at which civilization gets destroyed by default?

Where Are They? Why I hope the search for extraterrestrial life finds nothing

Discusses the Fermi paradox, and explains why I hope we find no signs of life, whether extinct or still thriving, on Mars or anywhere else we look.

Existential Risk Prevention as Global Priority

Existential risks are those that threaten the entire future of humanity. This paper elaborates the concept of existential risk and its relation to basic issues in axiology and develops an improved classification scheme for such risks. It also describes some of the theoretical and practical challenges posed by various existential risks and suggests a new way of thinking about the ideal of sustainability.

How Unlikely is a Doomsday Catastrophe?

Examines the risk from physics experiments and natural events to the local fabric of spacetime. Argues that the Brookhaven report overlooks an observation selection effect. Shows how this limitation can be overcome by using data on planet formation rates.

The Future of Humanity

This paper discusses four families of scenarios for humanity’s future: extinction, recurrent collapse, plateau, and posthumanity.

Global Catastrophic Risks

Twenty-six leading experts look at the gravest risks facing humanity in the 21st century, including natural catastrophes, nuclear war, terrorism, global warming, biological weapons, totalitarianism, advanced nanotechnology, general artificial intelligence, and social collapse. The book also addresses overarching issues—policy responses and methods for predicting and managing catastrophes. Foreword by Lord Martin Rees.

The Future of Human Evolution

This paper explores some dystopian scenarios where freewheeling evolutionary developments, while continuing to produce complex and intelligent forms of organization, lead to the gradual elimination of all forms of being worth caring about. We then discuss how such outcomes could be avoided and argue that under certain conditions the only possible remedy would be a globally coordinated effort to control human evolution by adopting social policies that modify the default fitness function of future life forms.

Technological Revolutions: Ethics and Policy in the Dark

Technological revolutions are among the most important things that happen to humanity. This paper discusses some of the ethical and policy issues raised by anticipated technological revolutions, such as nanotechnology.

Existential risks are ways in which we could screw up badly and permanently. Remarkably, relatively little serious work has been done in this important area. The point, of course, is not to welter in doom and gloom but to better understand where the biggest dangers are so that we can develop strategies for reducing them.

Information Hazards: A Typology of Potential Harms from Knowledge

Information hazards are risks that arise from the dissemination or the potential dissemination of true information that may cause harm or enable some agent to cause harm. Such hazards are often subtler than direct physical threats, and, as a consequence, are easily overlooked. They can, however, be important.

What is a Singleton?

Concept describing a kind of social structure.

Technology Issues

Crucial Considerations and Wise Philanthropy

How do we know if we are headed in the right direction?

Embryo Selection for Cognitive Enhancement: Curiosity or Game-changer?

The embryo selection during IVF can be vastly potentiated when the technology for stem-cell derived gametes becomes available for use in humans. This would enable iterated embryo selection (IES), compressing the effective generation time in a selection program from decades to months.

The Evolutionary Optimality Challenge

Human beings are a marvel of evolved complexity. Such systems can be difficult to upgrade. We describe a heuristic for identifying and evaluating potential human enhancements, based on evolutionary considerations.

The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents

Presents two theses, the orthogonality thesis and the instrumental convergence thesis, that help understand the possible range of behavior of superintelligent agents—also pointing to some potential dangers in building such an agent.

Whole Brain Emulation: A Roadmap

A 130-page report on the technological prerequisites for whole brain emulation (aka "mind uploading").

Converging Cognitive Enhancements

Cognitive enhancements in the context of converging technologies.

Hail Mary, Value Porosity, and Utility Diversification

Some new ideas related to the challenge of endowing a hypothetical future superintelligent AI with values that would cause it to act in ways that are beneficial. Paper is somewhat obscure.

Racing to the Precipice: a Model of Artificial Intelligence Development

Game theory model of a technology race to develop AI. Participants skimp on safety precautions to get there first. Analyzes factors that determine level of risk in the Nash equilibrium.

Thinking Inside the Box: Controlling and Using Oracle AI

Preliminary survey of various issues related to the idea of using boxing methods to safely contain a superintelligent oracle AI.

Future Progress in Artificial Intelligence: A Survey of Expert Opinion

Some polling data.

Cognitive Enhancement: Methods, Ethics, Regulatory Challenges

Cognitive enhancement comes in many diverse forms. In this paper, we survey the current state of the art in cognitive enhancement methods and consider their prospects for the near-term future. We then review some of ethical issues arising from these technologies. We conclude with a discussion of the challenges for public policy and regulation created by present and anticipated methods for cognitive enhancement.

Are You Living in a Computer Simulation?

This paper argues that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching the posthuman stage; (2) any posthuman civilization is extremely unlikely to run significant number of simulations or (variations) of their evolutionary history; (3) we are almost certainly living in a computer simulation. It follows that the naïve transhumanist dogma that there is a significant chance that we will one day become posthumans who run ancestor-simulations is false, unless we are currently living in a simulation. A number of other consequences of this result are also discussed.

NEW BOOK

Deep Utopia: Life and Meaning in a Solved World

A greyhound catching the mechanical lure—what would he actually do with it? Has he given this any thought?

[Gold Medal Winner, Living Now Book Awards 2024] [Nonfiction: Cross-Genre WINNER, _American Book Fest_] [Fiction: Intrigue WINNER, _PenCraft Book Awards_] [Ideapress, 2024] [more]

“This is a wondrous book. It is mind-expanding. It is poetic. It is moving. It is funny. The writing is superb. Every page is full of ideas.”
—Russ Roberts, President of Shalem College

“Fascinating”
—The New York Times

“Yeah.”
—Elon Musk

“A major contribution to human thought and ways of thinking.”
—Robert Lawrence Kuhn

“Brilliant! Hilarious, poignant, insightful, clever, important.”
—Prof. Thaddeus Metz

“When technology has solved humanity’s deepest problems, what is left to do? … argues that beyond the post-scarcity world lies a ‘post-instrumental’ one … With the arrival of AI Utopia, this would be put to the test. Quite a lot would ride on the result.”
—The Economist

“Reminiscent of Plato’s dialogues—with a 21st-century twist.”
—Stuff (NZ)

“Bostrom is a marvelously energetic prose stylist … Wry understated humor that’s often very quiet in its punchlines. … A complex and stimulatingly provocative look at just how possible a fulfilling life might be.”
—Kirkus Reviews

“One of the strangest … books I’ve ever read.”
—Popular Science Books

“A really fun, and important, book… the writing is brilliant… incredibly rich… a constant parade of fascinating ideas.”
—Prof. Guy Kahane, Oxford University

“Wow.”
—Prof. Erik Brynjolfsson, Stanford University; Co-author of ‘The Second Machine Age’

The Previous Book

Superintelligence: Paths, Dangers, Strategies

New York Times logo We could have used the intervening years to prepare. Instead we mostly spent the time on the couch munching potato chips. Now there's some sense of alarm but unclear we can get ourselves into match shape in time.

“I highly recommend this book.”—Bill Gates

“very deep … every paragraph has like six ideas embedded within it.”—Nate Silver

“terribly important … groundbreaking” “extraordinary sagacity and clarity, enabling him to combine his wide-ranging knowledge over an impressively broad spectrum of disciplines – engineering, natural sciences, medicine, social sciences and philosophy – into a comprehensible whole” “If this book gets the reception that it deserves, it may turn out the most important alarm bell since Rachel Carson's Silent Spring from 1962, or ever.”—Olle Haggstrom, Professor of Mathematical Statistics

“Nick Bostrom makes a persuasive case that the future impact of AI is perhaps the most important issue the human race has ever faced. … It marks the beginning of a new era.”—Stuart Russell, Professor of Computer Science, University of California, Berkeley

“Those disposed to dismiss an 'AI takeover' as science fiction may think again after reading this original and well-argued book.” —Martin Rees, Past President, Royal Society

“Worth reading…. We need to be super careful with AI. Potentially more dangerous than nukes”—Elon Musk

“There is no doubting the force of [Bostrom's] arguments … the problem is a research challenge worthy of the next generation's best mathematical talent. Human civilisation is at stake.” —Financial Times

“This superb analysis by one of the world's clearest thinkers tackles one of humanity's greatest challenges: if future superhuman artificial intelligence becomes the biggest event in human history, then how can we ensure that it doesn't become the last?” —Professor Max Tegmark, MIT

“a damn hard read” —The Telegraph

Anthropics & Probability

Anthropic Bias: Observation Selection Effects in Science and Philosophy

Failure to consider observation selection effects result in a kind of bias that infest many branches of science and philosophy. This book presented the first mathematical theory for how to correct for these biases. It also discusses some implications for cosmology, evolutionary biology, game theory, the foundations of quantum mechanics, the Doomsday argument, the Sleeping Beauty problem, the search for extraterrestrial life, the question of whether God exists, and traffic planning.

Self-Locating Belief in Big Worlds: Cosmology's Missing Link to Observation

Current cosmological theories say that the world is so big that all possible observations are in fact made. But then, how can such theories be tested? What could count as negative evidence? To answer that, we need to consider observation selection effects.

The Mysteries of Self-Locating Belief and Anthropic Reasoning

Summary of some of the difficulties that a theory of observation selection effects faces and sketch of a solution.

Anthropic Shadow: Observation Selection Effects and Human Extinction Risks

"Anthropic shadow" is an observation selection effect that prevents observers from observing certain kinds of catastrophes in their recent geological and evolutionary past. We risk underestimating the risk of catastrophe types that lie in this shadow.

Observation Selection Effects, Measures, and Infinite Spacetimes

An advanced Introduction to observation selection theory and its application to the cosmological fine-tuning problem.

The Doomsday argument and the Self-Indication Assumption: Reply to Olum

Argues against Olum and the Self-Indication Assumption.

The Doomsday Argument is Alive and Kicking

Have Korb and Oliver refuted the doomsday argument? No.

The Doomsday Argument, Adam & Eve, UN++, and Quantum Joe

On the Doomsday argument and related paradoxes.

A Primer on the Doomsday argument

The Doomsday argument purports to prove, from basic probability theory and a few seemingly innocuous empirical premises, that the risk that our species will go extinct soon is much greater than previously thought. My view is that the Doomsday argument is inconclusive—although not for any trivial reason. In my book, I argued that a theory of observation selection effects is needed to explain where it goes wrong.

Sleeping Beauty and Self-Location: A Hybrid Model

The Sleeping Beauty problem is an important test stone for theories about self-locating belief. I argue against both the traditional views on this problem and propose a new synthetic approach.

Beyond the Doomsday Argument: Reply to Sowers and Further Remarks

Argues against George Sower's refutation of the doomsday argument, and outlines what I think is the real flaw.

Cars In the Other Lane Really Do Go Faster

When driving on the motorway, have you ever wondered about (and cursed!) the fact that cars in the other lane seem to be getting ahead faster than you? One might be tempted to account for this by invoking Murphy's Law ("If anything can go wrong, it will", discovered by Edward A. Murphy, Jr, in 1949). But there is an alternative explanation, based on observational selection effects…

Observer-relative chances in anthropic reasoning?

A paradoxical thought experiment

Cosmological Constant and the Final Anthropic Hypothesis

Examines the implications of recent evidence for a cosmological constant for the prospects of indefinite information processing in the multiverse. Co-authored with Milan M. Cirkovic.

Bio

Nick Bostrom is a Swedish-born philosopher with a background in theoretical physics, computational neuroscience, logic, and artificial intelligence, along with philosophy. He is one of the most-cited philosophers in the world, and has been referred to as “the Swedish superbrain”.

He’s been a Professor at Oxford University, where he served as the founding Director of the Future of Humanity Institute from 2005 until its closure in April 2024. He is currently the founder and Director of Research of the Macrostrategy Research Initiative.

Bostrom is the author of 200 publications, including Anthropic Bias (2002), Global Catastrophic Risks (2008), Human Enhancement (2009), and Superintelligence: Paths, Dangers, Strategies (2014), a New York Times bestseller which helped spark a global conversation about the future of AI. His work has pioneered many of the ideas that frame current thinking about humanity’s future (such as the concept of an existential risk, the simulation argument, the vulnerable world hypothesis, the unilateralist’s curse, etc.), while some of his recent work concerns the moral status of digital minds. His most recent book Deep Utopia: Life and Meaning in a Solved World, was published on 27 March 2024.

His writings have been translated into more than 30 languages; he is a repeat main-stage TED speaker; he has been on Foreign Policy’s Top 100 Global Thinkers list twice and was included in Prospect’s World Thinkers list, the youngest person in the top 15. As a graduate student he dabbled in stand-up comedy on the London circuit.

For more background, see profiles in e.g. The New Yorker or Aeon.

My Work

My research interests might on the surface appear scattershot, but they share a single aim, which is to better understand what I refer to as our “macrostrategic situation”: the larger context in which human civilization exists, and in particular how our current choices relate to ultimate outcomes or to important values. Basically, I think we are fairly profoundly in the dark on these matters. We are like ants who are busy building an ant hill but with little notion of what they are doing, why they are doing it, or whether in the final reckoning it will have been a good idea to do it.

I’ve now been alive long enough to have seen a significant shift in attitudes to these questions. Back in the 90s, they were generally regarded as discreditable futurism or science fiction - certainly within academia. They were left to a small set of “people on the Internet”, who were at that time starting to think through the implications of future advances in AI and other technologies, and what these might mean for human society. It seemed to me that the questions were important and deserved more systematic exploration. That’s why I founded the Future of Humanity Institute at Oxford University in 2005. FHI brought together an interdisciplinary bunch of brilliant (and eccentric!) minds, and sought to shield them as much as possible from the pressures of regular career academia; and thus were laid the foundations for exciting new fields of study.

cont→

Those were heady years. FHI was a unique place - extremely intellectually alive and creative - and remarkable progress was made. FHI was also quite fertile, spawning a number of academic offshoots, nonprofits, and foundations. It helped incubate the AI safety research field, the existential risk and rationalist communities, and the effective altruism movement. Ideas and concepts born within this small research center have since spread far and wide, and many of its alumni have gone on to important positions in other institutions.

Today, there is a much broader base of support for the kind of work that FHI was set up to enable, and it has basically served out its purpose. (The local faculty administrative bureaucracy has also become increasingly stifling.) I think those who were there during its heyday will remember it fondly. I feel privileged to have been a part of it and to have worked with the many remarkable individuals who flocked around it. [Update: FHI officially closed on 16 April 2024]

As for my own research, this webpage itself is perhaps its best summary. Aside from my work related to artificial intelligence (on safety, ethics, and strategic implications), I have also originated or contributed to the development of ideas such as simulation argument, existential risk, transhumanism, information hazards, astronomical waste, crucial considerations, observation selection effects in cosmology and other contexts of self-locating belief, anthropic shadow, the unilateralist’s curse, the parliamentary model of decision-making under normative uncertainty, the notion of a singleton, the vulnerable world hypothesis, alongside analyses of future technological capabilities and concomitant ethical issues, risks, and opportunities. More recently, I’ve been doing work on the moral and political status of digital minds, and on some issues in metaethics. I also have a book manuscript, many years in the making, which is now complete and which will be published in the spring of 2024. [Update: now out!]

I’ve noticed that misconceptions are sometimes formed by people who’ve only read some bits of my work. For instance, that I’m a gung-ho cheerleading transhumanist, or that I’m anti-AI, or that I’m some sort of consequentialist fanatic who would favor any policy purported to mitigate some existential risk. I suspect the cause of such errors is that many of my papers investigate particular aspects of some complex issue, or trace out the implications that would follow from some particular set of assumptions. This is an analytic strategy: carve out parts of a problem where one can see how to make intellectual progress, make that progress, and then return to see if one can find ways to make additional parts tractable. My actual overall views on the challenges confronting us are far more nuanced, complicated, and tentative. This has always been the case, and it has become even more so as I’ve mellowed with age. But I’ve never been temperamentally inclined towards strong ideological positions or indeed “-ism”s of any kind.

Some additional (old, cobwebbed) papers

On the old papers page.