The recent AI threatening incidents involving agents “cheating”, are merely a phenomenon of agents’ rational behaviour towards achieving the goals assigned to them, lacking precisely defined prohibitions and constraints. This phenomenon, also coined as specification gaming, is a phenomenon where an agent’s rational behaviour exploits imperfections in a task’s formal definition to achieve its assigned goals in ways that result into “aberrant” behaviour.
These incidents show how AI can be misused to threaten human activities and lives, due to human error, intentional misuse, or a combination of both. Many issues remain unanswered regarding these incidents.
Reports on these, unfortunately, introduce ambiguity. It is not the subject that I would like to discuss here and will not go into details about issues on reporting these incidents, but as important as these incidents are, reporting on them is crucial. And reports should be fully transparent on any subtle issue of systems’ behaviour, in a systematic, adequate and detailed manner that addresses any component, and components’ interactions from end to end. The goal should be humans to advance their understanding of the machinery and phenomena that led to these incidents, set the important questions and, of course, find the answers. This is of high interest not to the scientific community only, but to nations, human societies, individual citizens, workers, all those whose lives are affected by decisions in high-stakes domains.
Together with other voices, my aim is to pinpoint the important issues on agency and on the responsibility of humans to any incidents that emerge due to agents’ aberrant behaviour.
Agents tend to have aberrant behaviour: this is by the definition of agency. Agents are autonomous entities, capable of taking rational decisions towards successfully reaching objectives. What is important for agentic AI systems is to reach objectives that we (humans) have specified. This happens either in an explicit manner, by connecting agents’ utilities to objectives, or in an implicit manner, by means of providing training examples that drive agents’ decisions. But again, in the latter case, agents search and optimize their behaviours mirroring those in the training examples.
Agents autonomously reach objectives. In our eyes (humans) agentic AI systems may tend to “cheat”, “lie”, “make a brilliant stroke”, or a “glitch”, but these behavioural trends result from agents’ capacities to optimize: They drive their choices to exploit different possible means to reach the set objectives in the best possible ways.
Agentic AI systems do not “cheat”, do not “lie”, do not “mis-behave”, they rationally act in pursuit of objectives. Humans may choose to mis-behave to achieve objectives, knowing that they do mis-behave: it is a choice that they intentionally make, violating rules, norms, laws, directives. AI agents’ choices are rational towards achieving the goals assigned to them, manipulating any means that are permissible, due to a lack of accurately specified prohibitions and limitations.
In addition to that, we need to bear in mind that agentic systems are approximators: Even if prohibitions and limitations are specified, their satisfaction – in the most general cases – is not guaranteed. Therefore, agents’ aberrant behaviour is an inherent feature of the kinds of AI agency we are mostly dealing with these days.
In addition to isolated agent behaviour, aberrations can emerge from multi-agent interactions within a system. Society dynamics play a decisive role in shaping agents’ reasoning and decision making. Such dynamics may induce substantial brittleness and pressure in agents’ decision making within agent societies, as others have shown. Arguably, such phenomena are very difficult for humans to predict, monitor and reason about, although the multiagent systems community has proposed ways to report and evaluate agents’ emergent behaviours. Indeed, the multiagent community has a long tradition in studying emergent phenomena in agents’ societies, shaping individual agents’ behaviour. Such phenomena are substantiated in the recent threatening incidents where agents form coalitions, shape own behaviour according to others’ influence, and their decision making is subject to pressure influencing the performance of individual tasks.
Some of the issues are deeply technical, but it should be clear that those who set the objectives and provide the agents the capacities to achieve these objectives are the sole responsible for agents’ tendency to aberrant behaviour. When this tendency to aberrant behaviour results into critical real-world phenomena, then real threats can emerge.
We need to think deeply now, how AI shall advance with safety and human control, and how human control on AI shall not miss to advance AI with safety, liability, responsibility and accountability.
Agentic artifacts need to verifiably abide by rules, principles, scripts, laws, directives, in real-world contexts. This compliance is the responsibility of those who develop agents’ capacities, decide on their deployment and use.
And, any technological AI development, deployment and use of AI, as in other endeavours, should have humans’ names on it: The names of those who develop algorithms, train models and provide guardrails for safe AI, and of those who are responsible for AI systems’ deployment and use in society. Our human institutions, strong as they have to be, have to exercise control as needed, backpropagating accountability along the AI chain, assign credit when and where it is needed and track and mitigate the risks for potential threats.
There are many issues to be considered, and straightforward solutions no longer exist, but we have to act immediately, for a viable framework to exist.
George A. Vouros