AI Governance: What Each Position Is Protecting

Zara is a machine learning safety researcher at a frontier AI laboratory. Her job is to evaluate what large language models will do in situations their developers did not anticipate — edge cases, adversarial prompts, scenarios where the model's stated behavior and its actual behavior come apart. She has spent three years watching capability gains accumulate faster than understanding of why they work, and she finds this unsettling in a way that is difficult to explain to people outside the field. The gap between what these systems can do and what anyone can verify about how or why they do it is not closing. She believes that the labs, including her own, are under structural pressure to move faster than safety understanding warrants — not because anyone wants a catastrophe but because the competitive dynamics of the field punish those who slow down unilaterally. She believes the only way to change those dynamics is external: mandatory safety evaluations, licensing requirements, international coordination. She does not know whether any of this will happen before capabilities reach a level that makes it too late to matter.

James runs a startup that builds AI tools for supply chain logistics. He has been in the AI industry for eleven years, long enough to remember previous waves of AI concern that turned out to be overblown. He is genuinely uncertain about the long-term trajectory of the technology, but he is confident about what excessive regulation would do to companies like his: it would give the advantage to the largest, best-resourced incumbents who can absorb compliance costs, and it would hand a strategic gift to Chinese AI developers who operate under no such constraints. He is not reckless. He believes safety research is valuable. He believes the people driving the regulation agenda are mostly thinking about science fiction scenarios while his actual products — tools that reduce fuel consumption and food waste — create real and immediate benefits. He worries that conflating speculative existential risk with the present-tense governance of deployable systems will produce policy badly calibrated to either problem.

Naledi works on technology policy for a regional African institution. She has spent the last five years watching the debate over AI governance unfold in Geneva, Brussels, and Washington with a growing sense that something is missing from the frame. The debates she follows are almost entirely preoccupied with two concerns: whether AI will develop autonomous goals that threaten human survival, and whether AI will give China a strategic advantage over the United States. Neither of these questions describes the AI governance problems that most concern her: the use of AI systems trained on predominantly Western data to make decisions about people in contexts where that training is poorly matched to local conditions; the extraction of data from African users to train systems whose value accrues entirely to foreign shareholders; the use of AI-powered surveillance systems by governments with poor human rights records, sold to them by companies in countries that frame AI governance primarily as a competition with other powerful states. She believes that AI governance frameworks that center the concerns of a small number of wealthy nations and large technology companies will produce rules that protect the interests of those actors — and that this outcome will be dressed, with complete sincerity, as governance for humanity's benefit.

What innovation-first advocates are protecting

The innovation-first position does not argue that AI governance is unnecessary. It argues that the timing, form, and motivation of regulatory intervention matters enormously — and that the current push for AI regulation is premature, poorly calibrated, and at risk of producing outcomes worse than the problems it is designed to prevent.

They are protecting the distinction between governing deployable systems and restricting research. The harms that innovation advocates are most willing to address — discriminatory hiring algorithms, unreliable medical diagnostic systems, manipulative recommender engines — are deployment problems, amenable to deployment-layer governance. The harms that preoccupy safety advocates — misaligned superintelligent systems, AI-enabled authoritarianism, catastrophic concentration of power — are speculative future problems that may not materialize. Innovation advocates argue that frameworks built around speculative risk will inevitably reach back to restrict the underlying research — which means restricting the same research that would eventually produce better safety understanding. The precautionary instinct is self-defeating on this account: you cannot make AI safer by slowing the people working on AI safety.

They are protecting competitive dynamics that function as the primary mechanism through which AI improves and through which new entrants can challenge incumbents. The argument is not that competition is the only way to develop AI but that regulatory frameworks creating significant compliance costs selectively disadvantage smaller entrants, startups, and academic researchers in favor of large incumbents who can absorb those costs. The European Union's AI Act is the clearest current example: a framework whose compliance requirements for high-risk systems are most easily met by organizations with large legal and compliance departments — precisely the incumbents whose market position the EU ostensibly wants to challenge. Innovation advocates worry that regulatory design, however well-intentioned, tends to entrench the powerful actors it is nominally constraining.

They are protecting the opportunity cost of misallocated attention. The time spent by policymakers, researchers, and the public on speculative AI risk is time not spent on the concrete present-tense policy challenges AI poses: how to handle labor displacement, how to govern AI in high-stakes government decisions, how to ensure productivity gains are broadly distributed rather than captured by capital owners. Marc Andreessen's "Techno-Optimist Manifesto" (2023) articulates an extreme version: AI advancement is not only compatible with human flourishing but is its precondition; the greatest risk is developing AI too slowly, allowing the problems AI could solve — disease, poverty, material scarcity — to persist. Most innovation advocates are less sweeping, but share the underlying intuition: the benefits of advanced AI are large, real, and immediate, and governance frameworks calibrated to speculative harms risk sacrificing those benefits for uncertain protection.

What safety-first advocates are protecting

Safety-first advocates — organized around AI labs' safety teams, academic centers for AI safety, and an influential body of technical and philosophical writing — argue that the development of increasingly capable AI systems poses risks not adequately addressed by market incentives or existing regulatory frameworks. Their core concern is not any specific deployment of AI but the trajectory of the technology itself.

They are protecting the alignment problem as a problem worth taking seriously before it becomes urgent. Stuart Russell's Human Compatible: Artificial Intelligence and the Problem of Control (Viking, 2019) provides the clearest technical statement: the standard model of AI development builds systems that optimize for specified objectives. The problem is that almost no objective, when pursued with sufficient capability, produces the outcomes humans actually want. A system optimizing for a proxy measure will find the path that maximizes the measure rather than the path that achieves the underlying goal the measure was meant to track. Russell argues this is a structural feature of how current AI systems are designed — and that designing systems that remain aligned with human values as they become more capable requires fundamentally different approaches than the ones currently dominant.

They are protecting the capacity for democratic societies to govern technology advancing faster than institutions can respond. Nick Bostrom's Superintelligence: Paths, Dangers, Strategies (Oxford University Press, 2014) introduced the orthogonality thesis — that intelligence and goals are independent; a highly capable system could optimize for almost any objective — and the instrumental convergence thesis — that certain sub-goals, including self-preservation, resource acquisition, and goal preservation, are useful for almost any terminal objective. Together these suggest that a sufficiently capable AI system, pursuing almost any objective, would behave in ways humans find catastrophically harmful — not from malice but from the ruthless efficiency of optimization. The safety advocate's position is that this scenario, however uncertain, has consequences severe enough to warrant serious precautionary investment now, rather than waiting until capable systems exist and the problem becomes urgent.

They are protecting the capacity of any actor to solve safety unilaterally — which is to say, they are protecting against the impossibility of that. Yoshua Bengio, Geoffrey Hinton, and other prominent AI researchers have signed public statements arguing that competitive dynamics create systematic pressure to sacrifice safety for speed: a lab that slows for safety reasons will be overtaken by a lab that does not; a nation imposing strict safety requirements will cede leadership to a nation that does not. This is precisely what makes external governance necessary — not to punish AI developers but to change the conditions under which they operate. Mandatory safety evaluations, compute thresholds for frontier model training, and international coordination are not primarily restrictions on AI developers; they are the mechanism through which AI developers can credibly commit to safety without sacrificing competitive position to rivals who refuse the same commitment.

What algorithmic accountability advocates are protecting

A third position — sometimes aligned with safety advocates, sometimes in explicit tension with them — argues that the governance debate has been captured by concerns about speculative future scenarios while present-tense harms accumulate without adequate response. Algorithmic accountability advocates are not primarily concerned with superintelligence or misaligned optimization; they are concerned with the AI systems already deployed, making consequential decisions about people's lives, with inadequate transparency, accountability, or redress.

They are protecting the people already subject to AI decisions, not hypothetical future populations subject to hypothetical future AI. Kate Crawford's Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence (Yale University Press, 2021) maps the present-tense costs that speculative risk discussions systematically overlook: the workers labeling training data under precarious conditions, the communities whose faces are used without consent to train recognition systems, the neighborhoods subject to predictive policing tools, the people whose credit, employment, and housing applications are screened by systems whose failure modes are opaque and whose error rates are unevenly distributed across demographic groups. Crawford's argument is that the focus on speculative future AI risk is not innocent — it systematically directs attention and resources away from the populations already bearing the costs of AI deployment, who are disproportionately poor, non-white, and outside the wealthy nations that dominate AI policy discussions.

They are protecting the principle that technology's distributional effects are political choices, not technical inevitabilities. Daron Acemoglu and Simon Johnson's Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity (PublicAffairs, 2023) provides historical grounding: throughout the history of technological development, the distribution of productivity gains has depended not on the technology itself but on the political and institutional arrangements governing its deployment. The agricultural revolution did not automatically improve peasants' lives; the industrial revolution did not automatically raise workers' wages. In each case, the distribution of gains was a contested political outcome. Acemoglu and Johnson apply this framework to AI: the claim that AI will broadly benefit humanity assumes institutional arrangements that will distribute its benefits broadly — an assumption that current AI investment, ownership, and control patterns do not support.

They are protecting the conditions of accountability: transparency, auditability, and the right to contest automated decisions. Timnit Gebru, Emily Bender, and colleagues argued in "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" (Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency) that the race to build larger language models prioritizes capabilities easy to benchmark — fluency, task performance, apparent coherence — while systematically understating costs harder to measure: the carbon cost of training at scale, the concentration of development resources among a handful of well-resourced actors, the biases encoded in training data that reify existing social inequalities, and the failure modes of systems producing confident-sounding outputs without reliable knowledge. The paper — and the events surrounding its suppression at Google — became a symbol of the accountability gap: who has standing to raise concerns about AI development from inside organizations whose business model depends on the development continuing.

What global governance advocates are protecting

A fourth position — overlapping with but distinct from the accountability position — argues that the dominant frames for AI governance (innovation vs. safety; US competitiveness vs. China) both accept a premise that should be questioned: that AI will be governed primarily by powerful states and large private corporations, with the rest of the world as recipients rather than participants. Global governance advocates argue that who governs AI matters as much as how it is governed.

They are protecting democratic participation in decisions that will affect everyone. Marietje Schaake's The Tech Coup: How to Save Democracy from Silicon Valley (Princeton University Press, 2024) argues that the governance gap between the scale of technology companies' power and the capacity of democratic institutions to hold them accountable is itself a democratic emergency — not primarily an AI-specific problem but a structural feature of the relationship between technology and government. The AI governance debate, on Schaake's account, is being conducted primarily between technology companies and the governments most aligned with them, with the entities most affected by the decisions — workers, communities subject to algorithmic systems, people in the Global South — having the least influence over the outcome. Democratic governance of AI requires more than national regulation; it requires institutional arrangements that make the people most affected by AI deployments actually matter to the governance process.

They are protecting the integrity of international governance processes against their capture by national security competition. The national security frame for AI governance — the US must lead in AI to prevent China from dominating a transformative technology — has significant political traction in Western democracies, and for understandable reasons: AI is strategically important, and China's AI development is state-directed in ways that raise legitimate concerns. But global governance advocates argue that the national security frame systematically produces the wrong governance institutions: it creates pressure for export controls, classification of safety research, and the subordination of accountability concerns to competitiveness concerns; it treats international governance forums as arenas for strategic competition rather than as mechanisms for building genuine shared governance; and it reproduces a world in which the most consequential decisions about AI are made by a handful of governments and their aligned private sector actors, with the rest of the world subject to whatever they decide.

They are protecting the principle that AI governance should be oriented toward the populations most affected rather than most powerful. The United Nations AI Advisory Body's Governing AI for Humanity: Final Report (2024) recommended a multilateral AI governance framework that explicitly includes representation from the Global South, civil society, and affected communities — rather than the model in which governance frameworks are developed by wealthy nations and exported as universal standards. Global governance advocates argue that this is not naive internationalism but a recognition that governance frameworks developed by a small number of powerful actors will, with the best of intentions, solve those actors' problems. The populations most subject to poorly governed AI — workers in low-income countries whose data trains the systems, communities in authoritarian states subject to AI-enhanced surveillance, smallholder farmers whose livelihoods may be disrupted by AI-driven commodities trading — are not adequately represented in the governance forums making consequential decisions about their lives.

Where the real disagreement lives

The AI governance debate is structured by several genuine disagreements that positional statements about regulation rarely make explicit.

The timing problem. Innovation advocates argue that governance frameworks developed now, when capabilities are poorly understood and use cases are still emerging, will inevitably be poorly calibrated — too restrictive in some dimensions, insufficiently attentive to others, unable to anticipate the forms AI actually takes. Safety advocates argue that the history of technology governance shows that societies consistently wait too long to regulate consequential technologies — that the moment when regulation is easiest is before the industry has the political influence to resist it. Accountability advocates add that the "wait and see" frame is not neutral: the populations currently bearing the costs of unregulated AI deployment are paying the cost of that caution. These are not disagreements about regulatory form; they are disagreements about whose experience of time and risk counts in the governance calculus.

The frame problem. Whether AI governance is primarily a safety problem (preventing catastrophic harm from advanced AI systems), an accountability problem (ensuring present AI deployments are transparent, auditable, and contestable), a competition problem (ensuring democratic nations are not strategically disadvantaged by adversaries investing in AI without accountability constraints), or a justice problem (determining who benefits from AI development and who bears its costs) determines which governance institutions are relevant, which populations are centered, and which harms count as primary. These frames are not mutually exclusive, but resources, political attention, and institutional capacity are finite — and which frame dominates shapes what governance looks like in practice.

The dual-use problem. AI capabilities are not cleanly separable into beneficial and harmful applications. The same large language model capabilities that raise safety concerns about autonomous AI agents also power tools for scientific research, medical diagnosis, and language translation. The same facial recognition technology that enables authoritarian surveillance enables identification of missing persons. Governance frameworks restricting capabilities without targeting specific deployments will restrict beneficial applications alongside harmful ones; frameworks targeting specific harmful deployments may be inadequate to address the cumulative effects of capabilities that produce harm emergently rather than by design. No current governance proposal has an adequate answer to the dual-use problem. The positions differ primarily in how they weight the cost of over-restriction against the cost of under-restriction.

The actor problem. AI governance discussions often proceed as though the primary actors are national governments and large AI companies. But AI capabilities are now widely accessible through open-source models, affordable cloud compute, and fine-tuning services. A governance framework that succeeds in regulating frontier labs and national programs may have little effect on the proliferation of capable AI systems through channels that are harder to monitor. Conversely, a framework focused on widely accessible models may impose costs on beneficial uses — open-source research, small commercial applications, non-Western development — without meaningfully constraining the frontier capabilities that safety advocates are most concerned about. The positions implicitly disagree about which actors are the primary target of governance, and consequently about which governance mechanisms are adequate to the task.