Longtermist institutional reform report

The problem

The future is not guaranteed. A global catastrophe such as a civilization-ending plague could destroy not only the current generation but, as a consequence, every generation that would have come afterward. If our generation causes or fails to prevent such a disaster, we will be robbing trillions of future people of happy lives. These future people thus have an incredibly large stake in our actions, but, as of now, no representation in the present.

Failing to prevent existential risks is not the only way that people living in the present could betray those in the future. A future of “astronomical suffering” is one in which huge numbers of people live existences of unimaginable pain and misery. Risks of astronomical suffering pose the threat of not just a barren future, but a future containing immense suffering: consider a world in which the worst things humans have ever done to each other are repeated, at a large scale and without hope of escape, until the death of the universe.

By contrast, the future ahead could also be unimaginably good. Our descendants’ lives could be as different from our own as our lives are from those of our hunter-gatherer ancestors. If we play our cards right, our grandchildrens’ grandchildren could experience a future without scarcity, war, or disease. In their place, our descendants could experience never-before-seen levels of joy and meaning.

If national and international governance can impact the likelihood and severity of negative outcomes for future generations, then reforming political institutions could be an important mechanism for impacting the long-term future— for helping to ensure that future generations avoid extreme suffering. We see two important and related ways in which our current institutions could be performing badly. The first is broad but perhaps very prevalent: political short-termism, the tendency of political institutions to prioritise near-term benefits over larger long-term benefits. The second is more specific and, to some extent, a symptom of the first: poor institutions and governance surrounding global catastrophic risks.

Few institutions in the present are looking out for the future. By working to ensure that they take future people into account, we can give our voiceless descendants a say in their fate.

Existential risks and future prosperity

In our report on Safeguarding the Future, we discussed risks that threaten “the premature extinction of sentient life or the destruction of its long-run potential.” Such a catastrophe need not kill everyone; it would be sufficiently bad to destroy humanity’s long-term potential. Other definitions are possible; Bostrom and Cirkovic consider risks “that might have the potential to inflict serious damage to human well-being on a global scale," and Avin et al. refer to "scenarios that could, in severe cases, take the lives of a significant portion of the human population, and may leave survivors at enhanced risk by undermining global resilience systems."

All capture the core of the problem: scientific progress has reached the point where certain human-made technologies have extremely destructive capacities that could curtail the vast potential of the future of humanity. In The Precipice: Existential Risk and the Future of Humanity, Toby Ord surveys the scientific literature and concludes that the most pressing risks are posed by advanced artificial intelligence that is unaligned with human values and developments in biotechnology that enable easily engineered pandemics (by accident or by design). The range of possible existential risks extends to include extreme climate change scenarios, risks of severe nuclear weapons exchanges, and dangerous unforeseen future technological developments.

Probabilistic estimates of global catastrophic risks vary. Ord, for instance, gives a 1 in 10 chance (10%) of an existential catastrophe this coming century due to unaligned artificial intelligence and 1 in 30 chance (~3%) due to engineered pandemics. Garfinkel, by contrast, puts the risk of existential catastrophe from artificial intelligence in the 0.1% to 1% range this century.

While the most severe global catastrophes threaten the existence of future generations, lesser catastrophes and poor governance also threaten their prosperity and well-being. The sheer size of humanity’s prospective future, and the fact that that future depends on the compounded results of everything that has occurred before it, suggest that even small trajectory changes can have huge moral importance. For this reason, thinking hard about what trajectory changes might be most beneficial is one path toward maximizing the value of the future.

Can the future be influenced?

The world’s most important institutions control policy and governance levers that affect billions of current people and many more future ones. Motivating this research project is the concern that the future is potentially very valuable, and that ensuring that these institutions act in the best interests of future people is one way to maximize its potential.

We take a broad definition of “institutions” to include established rights, rules, norms, shared understandings and decision-making processes. We are interested here in political institutions in particular, which are the institutions that constrain and prescribe political actors’ interactions and decisions.

The simplest reason to believe that we can affect the future is that the present is still being affected by the distant past. Research has found at least two ways institutions can affect the long-run future: first, through self-perpetuation and second, by creating substantial changes in the societies they affect which persist through years, decades, or centuries.

In the first case, evidence from political science and economics suggests that economic policy can last despite significant changes in local conditions. Institutions set up in the distant past can exist into the present, or nearly so, largely unchanged. The Icelandic Althingi—the oldest surviving parliament in the world—was founded in 930 and met regularly for nearly a thousand years. The recently-shuttered Kongō Gumi, a Japanese temple-builder of great renown, was launched in 578 and managed by its founder’s descendants for more than 1400 years. Al-Azhar University has been the preeminent center of Arabic and Islamic learning since its foundation in 970. These examples suggest that, in rare cases, institutions can survive on exceptionally long timescales and pursue their original missions for the duration.

In the second case, researchers have demonstrated a wide variety of connections between historical and present-day outcomes. The economic literature on persistence suggests persuasively that economic development in the present can be driven by historically contingent factors centuries or even millennia in the past. Areas that had older state institutions in antiquity today have higher levels of income, all things equal. Putterman and Weil have demonstrated that ethnic groups originating in regions with longer histories of organized states tend to have higher levels of income after migrating to other regions. And researchers have found that public goods such as roads provided in the Roman Era may influence incomes today. For a more thorough review of the persistence literature, see Voth, Persistence--Myth and Mystery (2020).

The current state of long-term governance

We see two important and related ways in which our current institutions could be under-performing when it comes to safeguarding and benefiting the long-term future. The first is broad but perhaps very prevalent: political short-termism, the tendency of political institutions to prioritize near-term benefits over larger long-term benefits. The second is more specific and, to some extent, a symptom of the first: poor institutions and governance surrounding existential risks.

Political short-termism

Political short-termism is characterized by political processes that give priority to near-term benefits over larger long-term benefits. Governments run in the present are elected by and accountable to people currently alive. In many cases, however, their incentives act on a much shorter time scale. In particular, elected officials work in relatively short election cycles, lasting just a few years at most, which creates short-term incentives for re-election. As a result, it is plausible that many policy and decision makers are concerned primarily with the impacts of their actions on the scale of just a few years.

Political short-termism can have its origins in a number of separate pathologies. Roughly, these causes can fall into three categories: epistemic, motivational, and institutional determinants.

Broadly speaking, epistemic determinants are features of political actors’ state of knowledge that prevent even properly-motivated actors from adopting appropriately long-termist policy. This leads these actors to discount the future impacts of present actions.

Motivational determinants are those features of political actors’ goals and motivations that lead even well-informed actors to wrongfully discount the future. These drivers include a positive rate of pure time preference, pure self-interest, or relational partiality, in which political actors are motivated to benefit those they have close relations with, such as friends, family, or their community, at the expense of both current future generations.

Institutional determinants of political short-termism are features of political actors’ institutional context that strip the political means from otherwise well-informed and properly motivated actors who would otherwise adopt more appropriately long-termist policy, or which make political actors either less well-informed or less well-motivated. These determinants include:

• Election incentives that cause actors to prioritize policy with near-term results and visible benefits for which politicians and parties can take credit
• Economic dependency on various firms and other bodies that exert pressure on political actors to use short auditing durations, e.g. through performance indicators with short-term goals or short budget-windows
• A short media cycle that requires political actors to react and respond swiftly to political issues
• Political polarization that detracts from careful, collective deliberation and causes basic factual questions to become areas of political competition

Political short-termism plausibly leads to policy and decision-making that is far from optimal for long-term outcomes. This suggests that reforms that target these root causes may be very important. While the above considerations seem plausible to us, it is very challenging to establish the value of broad improvements to political institutions. We are uncertain about the extent to which the above factors drive political short-termism, how much political short-termism reduces the value of the long-term outcomes, and how mitigating the above factors would change policy outcomes.

Paths for managing existential risks

There are several things that governance systems could do (1, 2, 3, 4, 5):

1. Foresee – identifying global catastrophic risks ahead of time
2. Understand – understanding the risks we have identified in order to design measures to prevent, prepare for, respond to and recover from global catastrophes; analyzing potential global catastrophes; estimating the severity and probability of global catastrophic risks
3. Prevent – preventing global catastrophes that otherwise would have occurred
4. Prepare – designing systems to respond to and recover from catastrophes effectively
5. Respond – when catastrophes occur, acting to reduce the severity
6. Recover – building resilience to global catastrophes, so that we can effectively recover from catastrophes

In order to keep the risks and effects of global catastrophe at a sustainably low level, it is plausible that we will have to perform well in each of the above areas. Foreseeing and understanding existential risks is essential for managing and mitigating them later on. Preventing catastrophes will often be far preferable to responding to and recovering from them as the cost of prevention will often be much less than the cost of non-prevention. But if prevention fails, it is essential that we are well-placed to respond and recover. Even if we could reliably prevent all foreseeable global catastrophes, unforeseeable ones would still pose a significant risk, and so it is very important to build resilience to a wide range of risks.

How well are current institutions doing in these areas? The recent COVID pandemic offers one disheartening data point. Prior to the pandemic, the United Kingdom was one of the putatively best-prepared nations. Yet its performance throughout the crisis has been less robust than hoped for. How would the world have fared if faced with an even more contagious and deadly pathogen?

Solutions and theory of change

We turn now to investigating potential solutions to the problems raised in the previous section and in doing so, build a high-level theory of change for longtermist institutional reform. We begin by describing a long-term vision of a future trajectory we can hope to realize. Given this vision, we consider what we can do in order to achieve it. This results in a collection of intermediate objectives that will plausibly take us towards our long-term vision. From this, we derive a collection of short-term objectives, which are objectives that plausibly both (i) are achievable in the near future and (ii) could drive us towards the intermediate objectives. We then map out several interventions within the space of longtermist institutional reform and their causal pathways to the short-term objectives. Finally, we consolidate the findings of this section into a high-level theory of change.

Long-term vision

The dangers are many: a future of unimaginable suffering or airtight authoritarianism; a future in which the sand in humanity’s hourglass has run out and true catastrophes have conspired to ensure that there is no future at all. We envision the opposite: a future world in which the risk of global catastrophes is sustainably low and in which the interests of all are protected and advanced by institutions that have humanity’s long-term flourishing in mind. In particular, we imagine the mirror image of the dark future just conjured: a future of previously unseen happiness and meaning, a world in which all creatures experience unprecedented levels of safety, security, and well-being. These are the stakes.

With an annual chance of extinction of just 0.1%, the chance that humanity survives 1,000 years is less than 37%.¹ Even a low chance of catastrophe is unsustainable over very long time horizons so the best future is one in which the chance of extreme catastrophe is sustained at a very low level. Note that simply clearing this low bar is not our only goal: we aspire to a future in which humanity not only survives, but thrives.

It is important to recognize that the long-term vision is not merely a vision for what we want the future to look like at some point in time. It is rather a vision for what we want the future trajectory to look like for as long as possible. The vision is not merely to achieve protection from the greatest risks and a high quality of life, but to sustain this protection and quality of life for the long-term and, if possible, to continue to improve it over time. Overall, our long-term vision (LTV) consists of the following:

• LTV1. Continued existence of human civilization through sustainably low severity and probability of existential risks
• LTV2. Sustained protection from harm and suffering for all conscious creatures
• LTV3. Sustained inclusive progress for all conscious creatures

Intermediate objectives

We now turn to intermediate objectives that could be important for achieving our vision of long-term impact. These intermediate objectives are not intended as necessary and sufficient conditions for achieving the long-term vision. Rather, they are intended as suggestions for the kinds of political institutions we might try to reach in the coming decades, on the way to the long-term vision. The first three intermediate objectives (IOs) relate to the management of GCRs and the last two relate to protecting and advancing the interests of all.

Addressing existential risks

Recall the broad categories of action that we could take in order to manage existential risks: Foresee, understand, prevent, prepare, respond, and recover.

From this list, we can draw the first three intermediate objectives that we would like to achieve in the coming decades or sooner:

• IO1. Proactive institutions that foresee, understand, and prevent existential risks
• IO2. Well-prepared institutions that plan and prepare for ongoing, emerging and unforeseen existential risks
• IO3. Resilient institutions that effectively mitigate catastrophes, preventing or reducing the severity of and recovering from global catastrophes

The achievement of these intermediate objectives increments the probability that humanity will survive by lowering the probability or the severity of existential risks. Though it is not logically necessary for all intermediate objectives to be achieved in order to fulfill this goal, it is difficult to imagine a secure world otherwise. In particular, if IO3 is not achieved, then the degree to which the other two objectives must be fulfilled is necessarily extremely high.

Protecting and advancing the interests of all

One of the most unpleasant scenarios to imagine for humanity’s future is one in which extreme suffering is a permanent fact of life, facilitated perhaps by advanced technology or the lock-in of totalitarian institutions. Caring for the long-term entails ensuring that we can avoid the pathologies that have plagued human civilization for millennia: the exploitation and marginalization of large groups of people and the imposition of the will of a small group by overwhelming force. These suggests two additional intermediate objectives:

• IO4. Protection from harm and exploitation through well-functioning, inclusive institutions
• IO5. Inclusive progress through well-functioning, inclusive political institutions

If these intermediate objectives can be approached via robust, long-lived institutions, long-term vision elements two and three (LTV2 & LTV3) can be securely achieved.

Short-term objectives

We now consider the short-term objectives that we should aim for in the coming years and decades. These are objectives that should help get us to the intermediate objectives. We start by considering solutions to political short-termism and how these could take us towards the intermediate objectives.

Solving political short-termism

Full solutions to political short-termism will need to address each of the determinants of political short-termism (epistemic, motivational and institutional). In short, through institutional reforms, we need to make policy makers better informed and more motivated to take long-term consequences into account and to remove institutional barriers to long-term governance.

In order to address the epistemic determinants, we need the relevant knowledge to be integrated into key long-term governance systems, such as national governments and important international organizations including the United Nations. However, there are significant open questions as to what the “relevant knowledge” is. It is therefore essential both that learning on these topics continues and that the resulting knowledge is communicated credibly to policy makers so that policy makers integrate it into their reasoning and decision making processes.

Effective, credible communication of important lessons (new or old) could have implications for the motivational determinants of political short-termism in two ways. Firstly, such communication could directly motivate policymakers to take the long-term into account to a greater extent, for example, by making the needs of future generations more salient. Secondly, such communication could influence the opinion of policymakers’ constituents in a similar fashion, providing indirect incentives for policymakers to take the long-term into account to a greater extent. More broadly, the motivational determinants of political short-termism may be best addressed by improving the political incentives around long-term governance, either by addressing the institutional determinants of political short-termism or indirectly (e.g. by influencing public opinion). In particular, we would like to see stronger incentives to assess future policy impacts and, especially, to manage existential risks.

One objective we could aim for that could be particularly important is the political inclusion of future generations, for example, through commissioners or parliamentary committees tasked with representing the interests of future generations. Providing political power to future generations would create incentives for policy makers to take their interests into account, to plan better for the long term and to manage existential risks. Focusing on future generations could be especially powerful as problems for people can be much more salient than abstract concepts like “the long-term future” and existential risks.

Recent successes in this regard include the formation of future-focused government bodies in Finland, Singapore, and the United Kingdom.

Managing existential risks in the near term

Recall the intermediate objectives around existential risks. We have already seen that we need improved knowledge around the long-term and existential risk in order to address the epistemic determinants of political short-termism. We need improved learning and understanding around existential risks and to integrate this knowledge into key institutions through credible, effective communication, in order to shape the proactive institutions described in IO1.

Enabling strong preparation and resilience as described in IO2 and IO3 will require subject-specific knowledge, capability and expertise. These considerations suggest that the following short-term objectives (STOs) could be useful targets on the way to achieving IO1-3:

• STO1. Improved learning and understanding around long-term governance existential risks.
• STO2. Credibly spreading learnings and understanding around long-term governance and existential risks.
• STO3. Improving and integrating long-term governance and knowledge and expertise about existential risks within important governance organizations

Protecting and advancing the interests of all

Recall the intermediate objectives (IO4 and IO5) around protecting and advancing the interests of all. The short-term objectives STO1-3 will contribute to these objectives to the extent that they reduce political short-termism and to the extent that such reductions in political short-termism protect and advance the interests of all. However, STO1-3 will likely be insufficient to achieve IO4-5: many of the motivational and institutional determinants of political short-termism remain. We suggested that addressing these will require stronger institutional incentives for policy makers to take the long-term into account and that a particularly promising objective in this vein was the political inclusion of future generations. As a result, we suggest the following short-term objectives:

• STO4. Improving the political incentives around long-term governance to foster greater long-term thinking in important governance organizations
• STO5. Greater political inclusion of future generations within important political institutions

Theory of change

We can combine the above objectives and long-term vision to form a high-level theory of change:

Note that many of these objectives are interlinked, and that this diagram does not incorporate the many separate pairwise linkages between short-term, intermediate, and long-term objectives and impacts. This is, rather, a “high-level” theory of change. We flesh this out in the figure below by mapping out the specific activities we could carry out within the space of longtermist institutional reform and showing how they could help us achieve the short-term objectives.

Specific activities

Potential harms

The above theory of change is intended to help identify what we need to do in order to have a positive long-term impact. In order to come to a more complete view of the kinds of activities we should be carrying out, it is also important to think about how we could have a negative long-term impact.

The most obvious risk of harm is the direct avenue: harm can occur if an intervention actively increases the risk of global catastrophe, reduces concern for future generations, or weakens institutions’ ability to respond to threats and govern effectively. One way that an intervention could lead to direct harm is via so-called “information hazards”: by propagating data about global catastrophic risks, organizations could potentially raise the probability that malicious actors will take steps that will exacerbate those risks.

Another potential risk is that of politicization. At present, the comparative lack of focus on existential risk and long-term governance means that relevant issues are not yet highly politicized. This means not just that such issues are not a major element of political debate, but also that key positions have not been staked out and associated with competing political parties. If this changes—for instance, if concern for future generations becomes associated with one major American political party—long-term governance and existential risk avoidance could become much less tractable.

A final key possibility of harm is that of crowding out: if money in the longtermist institutional reform space is spent unwisely, it could raise the profile of less-effective organizations to such an extent that these efforts displace more effective ones. In this scenario, a poor choice of investment causes harm by depriving useful efforts of much-needed attention and money. A secondary failure mode in this category is one in which less-effective “angles” on governance or existential risk avoidance capture a large amount of research and/or attention, leading high-leverage avenues to languish unfunded.

Key prioritization considerations

The previous section provides a map of the specific activities we could carry out within the space of longtermist institutional reform. In combination with earlier sections, this sketches how such activities could have a long-term impact. We now turn to the important question of which activities we should currently prioritize with additional funding.

Urgency of existential risk reduction

Questions about how to spend money to benefit the long-term future hinge significantly on those about the urgency of existential risk reduction. Opinions differ as to how urgent various risks are. If we believe that certain catastrophes threaten to wipe humanity out in the very near term, then the concern is urgent and we should prioritize institutional reforms that could lead to a rapid reduction in near-term existential risk. By contrast, if risk persists at a relatively low level, then the concern is less urgent and humanity is best-served by waiting and learning more; this pushes in the direction of broad long-term governance improvements.

An equally important concern is the degree to which different potential disasters pose different levels of risk. Each prospective existential risk presents different considerations for potential funders: all else equal, we should prioritize working on the most urgent risks, but those are not necessarily those where an additional dollar spent by a philanthropist will go farthest.

Uncertainty

Overall, our uncertainty about which risks to prioritize—and how to prioritize different ways to mitigate them—pushes in the direction of flexible interventions that can address multiple threats. Likewise, our uncertainty about the urgency of different existential risks in general suggests that we should consider funding broader long-term interventions as well. Once we have established broad areas in which we want to look for funding opportunities, we face two key, additional uncertainties:

• Uncertainty about what works (and what doesn’t)
• Uncertainty about what might be unintentionally harmful

Focusing on learning and credibility-building helps to mitigate some uncertainty because such activities come with a low risk of harm. But a more compelling case for focusing on these activities lies in their relative cost-effectiveness. Given the astronomical scales involved in long-term considerations, delaying action by a few years is a small price to pay for much better, research-informed policy—especially when avoidance of harm is a key consideration.

Geography

When prioritizing potential funding opportunities in the area of longtermist institutional reform, geography is another key consideration. Because global catastrophic risks are, by definition, global problems, solutions that act on a global scale are preferred. This is not as simple as addressing potential solutions to major international institutions: such institutions may have weaker influence on the geographies they govern than do national governments; moreover, some national governments (such as the US) can have considerably more influence than international bodies.

One potential avenue to encouraging global solutions without acting directly on global institutions is to consider that some small countries can have outsized influence as role models or policy exporters. If it is easier to implement long term-focused policy in these smaller jurisdictions—and there is some evidence that this may be the case—doing so may be of large marginal value.

Other reasons to consider prioritizing work in smaller countries relate to harm avoidance (it is more difficult to cause large harms when working with less powerful governments) and tractability (in small policy networks, there is less competition and it may be easier to quickly get into an influential position). Note, however, that tractability depends not just on features of the geography but also on the extent to which there are people or organizations in the region who are able to do good work. Finding such people or organizations is a key task for potential grant-makers in this area.

Conclusions

We believe that learning and credibility building around existential risks seem among the most likely to have a positive impact and least likely to have a large negative impact (especially through research; policy experimentation is riskier).

Moreover, we believe that working in small, well-functioning countries is both lower-risk and potentially more tractable. How valuable this work is depends on (a) how likely it is that such countries will be able to export good policies elsewhere, (b) the urgency of different risks, and (c) how much we could learn from working in such countries. We note, however, that the upside potential offered by building credibility and policy support in larger, more powerful institutions and governments can outweigh these tractability concerns.

We also conclude that improving long-term governance and managing existential risks will both likely require improvements on national and international levels. This suggests we should be open to working on both axes; whether good funding opportunities can be found in each setting is an open question.

Finally, we conclude that improved long-term thinking and governance could plausibly be useful by enabling small trajectory changes that compound over time. If we think humanity’s future trajectory is very path-dependent in a way that depends on what our national and international governance systems look like, then making these systems more capable at long-term governance could be very good. Relatedly, the political inclusion of future generations seems to us to be of plausibly high value: this inclusion constitutes a potentially useful platform and tool for existential risk reduction and for improving long-term governance.

Notes

1. 0.9991000 = 0.368. This calculation assumes that the chance of extinction in each year is independent of all the other years. ↩