The Illusion of Machine Agency: Why Treating AI Agents as Autonomous Decision-Makers Threatens Corporate Accountability
As artificial intelligence rapidly transitions from passive advisory tools to active, self-directed systems capable of executing complex workflows, a critical debate has emerged over the boundaries of machine autonomy and human responsibility. In its fifth consecutive year of collaboration, the MIT Sloan Management Review and Boston Consulting Group (BCG) have convened an international panel of over 50 industry practitioners, legal scholars, academics, and policymakers to examine the implementation of responsible artificial intelligence across global organizations.
The latest inquiry from this ongoing initiative addresses a provocative central premise: that responsible governance treating AI agents as autonomous decision-makers is fundamentally flawed and destined to fail. While an overwhelming 72% of the expert panel initially agreed with the statement, subsequent qualitative analysis reveals a far more complex reality. The consensus suggests that while AI agents exhibit profound operational and technical independence, characterizing them as "autonomous decision-makers" creates a dangerous governance vacuum. This linguistic framing risks enabling human creators, corporate deployers, and institutional stakeholders to evade legal and moral liability when automated systems cause real-world harm.
Background and Context of the AI Agency Shift
The contemporary landscape of enterprise technology is increasingly dominated by "agentic AI"—systems engineered not merely to answer user queries or synthesize text, but to autonomously plan, invoke software tools, execute transactions, and navigate multi-step workflows. Unlike traditional automation scripts, these modern AI agents operate with significant operational latitude, determining their own sequences of actions to achieve user-defined goals within established parameters.
This evolution has accelerated over the past several years, driven by breakthroughs in foundational large language models, multimodal capabilities, and integrations with enterprise resource planning software. However, as these systems permeate core business functions—ranging from inventory management and risk pricing to customer service and financial transactions—they have outpaced existing regulatory frameworks and internal corporate compliance standards.
The tension between technical capability and institutional accountability gained mainstream legal prominence through cases such as the landmark British Columbia Civil Resolution Tribunal ruling in Moffatt v. Air Canada. In that dispute, the tribunal flatly rejected Air Canada’s defense that its customer-service chatbot was a distinct legal entity responsible for its own misstatements, reinforcing the principle that corporations remain strictly liable for the automated representations and actions of their software. The MIT Sloan Management Review and BCG panel initiative builds upon these mounting legal and operational challenges, offering timely insights into how organizations must adapt their governance models to address the rise of agentic AI.
Expert Insights and the Autonomy Paradox
The panel discussion underscores a vital semantic and structural distinction: operational autonomy does not equate to moral or legal accountability. Experts contributing to the initiative highlighted the perils of attributing true agency to software.
Rainer Hoffmann, chief data officer at EnBW, noted that while "agentic real-world autonomy is real and growing," Renato Leite Monteiro, vice president of privacy, data protection, AI, and intellectual property at e&, cautioned that "self-improving agents are moving faster than we can map their failure modes." Similarly, Ben Dias, chief AI scientist at IAG, observed that agentic AI is rapidly transcending passive support to take direct autonomous action on behalf of users. Simon Chesterman, vice provost at the National University of Singapore, emphasized that these systems can plan, execute, and transact across complex workflows, leading AI speaker and consultant Linda Leopold to characterize them as technically autonomous because they operate without constant human oversight.
However, legal and ethical scholars on the panel pushed back against expanding this technical definition into moral domains. Bruno Bioni, founder and director of Data Privacy Brasil, argued that what frequently passes for autonomy is fundamentally "delegated execution"—the selection of steps and utilization of tools strictly within pre-defined operational boundaries set by humans. Amit Shah, CEO of Instalily.ai, echoed this sentiment, describing agents as infrastructure that decides in a purely operational sense by routing orders and pricing risk, while noting that "a machine can make the call, but it cannot own the outcome or consequence in the moral sense."
Legal experts warned of the severe systemic risks introduced when organizations conflate technical independence with legal personhood. Riyanka Roy Choudhury, a Stanford CodeX fellow, asserted that treating agents as autonomous decision-makers "severs liability from capacity," given that software holds no corporate assets, maintains no licenses, and possesses no deterrable interests. Öykü Işık further observed that stochastic, context-dependent AI agents lack the stable intent required for meaningful accountability, making the designation of "autonomous decision-maker" a hazardous governance fiction.
The Danger of Blame Laundering
A primary concern raised by the panel is the emergence of an "accountability vacuum." When executives, developers, and operators frame AI systems as autonomous decision-makers, they risk creating a mechanism for what critics term "blame laundering."
Simon Chesterman warned that adopting language where agents "decide" allows firms and governments to deflect responsibility seamlessly through the machine—summarized by the cyclical excuse that the model recommended, the agent acted, and the human shrugged. Bruno Bioni added that this vocabulary permits developers and deployers to retreat behind the refrain of "the AI decided" whenever operational outcomes go awry. For enterprises that remain legally and financially accountable for the actions of their software tools, cultivating this culture of displaced blame introduces existential compliance and reputational risks.
Stratifying Risk and the Limits of Delegation
Despite widespread agreement on the dangers of unbridled agency, the panel acknowledged that governance must be proportional to risk. Not all AI applications carry the same weight, and operational autonomy can be appropriately calibrated based on the context of deployment.
Richard Benjamins, co-CEO of RAIght.ai, suggested that while impactful decisions should never be fully outsourced to autonomous AI, trivial or low-stakes decisions can be safely managed with minimal human intervention. Katia Walsh, AI lead at Apollo Global Management, and Carolina Aguerre, a professor at Universidad Católica del Uruguay, both emphasized that the level of autonomy granted to an agent must be rigorously assessed against the specific tasks, objectives, and risk profiles involved. As consultant Pierre-Yves Calloc’h succinctly concluded, responsible AI practice requires knowing precisely where autonomy must stop.
Analyzing the Sociotechnical System
To resolve the governance dilemma, experts argue that organizations must look beyond the isolated software model and evaluate the entire sociotechnical ecosystem. Belona Sonna of the Australian National University noted that in domains like autonomous driving, real-time operational demands require machine-led decisions, making the central challenge not autonomy itself, but ensuring that behavior remains aligned with human values and ethical principles.
Mark Surman, president of Mozilla, stressed that AI agents do not materialize in a vacuum; they are built by developers, deployed by enterprises, and monetized by stakeholders. Consequently, governance frameworks must treat agents as extensions of human and institutional choices. Stefaan Verhulst, chief research and development officer at GovLab, reinforced this perspective, arguing that oversight mechanisms must recognize agents as active participants within broader institutional, legal, and cultural frameworks.
Actionable Recommendations for Enterprise Leaders
Drawing from the collective insights of the international expert panel, organizations seeking to responsibly integrate operationally autonomous agents are advised to implement five foundational strategies:
-
Calibrate Autonomy According to Stakes, Not Capabilities
Organizations must decouple technical capability from deployment permission. Just because an agent possesses the technical capacity to execute a workflow independently does not mean it should be authorized to do so. Delegation decisions must be strictly based on the reversibility, real-world impact, and potential risk of each action, with dynamic thresholds that adjust as stakes evolve. -
Enforce Limits by Design, Not Policy Alone
Written compliance guidelines and prompt-based instructions are insufficient to control agentic behavior. Enterprises must bake operational boundaries, approval gates, hard stops, and scoped technical permissions directly into the system architecture to prevent unauthorized actions at the code level. -
Assign Explicit Human Accountability Before Deployment
Regulatory bodies, courts, and corporate boards require identifiable human owners for all automated outcomes. Organizations must assign unambiguous accountability for agent activities to specific roles or individuals prior to deployment, ensuring that cross-functional siloes maintain clear oversight, escalation, and monitoring responsibilities. -
Govern the Ecosystem, Rather Than the Model
Accountability structures must target the entire sociotechnical lifecycle rather than focusing narrowly on the AI model or agent. Effective governance encompasses the developers who designed the system, the enterprise that deployed it, the individuals who authorized its scope, and the operating context, ensuring responsibility remains firmly anchored within the human workforce. -
Foster a Culture of Challenge and Accountability
As human-agent collaboration increases, organizational cultures must empower employees to scrutinize, challenge, and override agent outputs without fear of professional reprisal. Establishing clear documentation of shared responsibilities—especially when integrating external vendor-supplied agents—mitigates ambiguity and strengthens overall organizational integrity.
About the Program and Authors
The Responsible AI initiative is an ongoing research program by MIT Sloan Management Review and Boston Consulting Group, utilizing global executive surveys and curated expert panels to deliver actionable insights on emerging governance standards. The initiative is led editorially by Elizabeth M. Renieris, contributing editor and senior research associate at Oxford’s Institute for Ethics in AI; David Kiron, editorial director of research at MIT Sloan Management Review; alongside BCG managing directors and partners Steven Mills, chief AI ethics officer, and Anne Kleppe, global lead for responsible AI.
