Agency, Transparency, and Practicality
Koh Achim and Cheon Hyundeuk
Image: Professor Cheon Hyundeuk’s laboratory, Seoul National University, July 16, 2025. Photo courtesy of: Sooyoung Leam
November 14, 2025
Koh Achim (KA): You often emphasize that any discussion of AI ethics and concepts such as transparency and explainability must begin with a substantive inquiry into what AI actually is. While it may seem obvious that such a discussion presupposes an understanding of AI itself, I would like to press the issue more concretely. Philosophically, what does it mean to determine the very “nature” of AI?
Cheon Hyundeuk (CH): I believe that ethics must be grounded in ontology and epistemology. Some philosophers in the Continental tradition, broadly speaking, have argued that ethics should come first, with epistemology and metaphysics deriving from it.[1] My view, however, is that ethical inquiry becomes viable only after we examine how a given technology has developed and we clarify its essential character.
Consider virtual reality, for example. When the boundary between the real and the virtual becomes blurred, one might argue that the distinction between reality and virtuality/fictionality has collapsed and that everything now counts as reality. I would nonetheless maintain that the distinction remains crucial, even if the criteria for it have shifted. By the same token, at our present technological juncture, we must first determine the ontological status of AI. Only then can we address the ethical questions it raises, especially in times of such conceptual confusion.
At the Center for Ethical, Legal and Social Issues in AI (AI ELSI Center), of which I am a director, we recently debated whether AI could possess consciousness. Even if it currently cannot, the question remains crucial, since consciousness serves as a criterion for moral consideration. For instance, a person in a vegetative state is deemed to have no consciousness—or a severely diminished level of it—and their rights and treatment are altered accordingly. Therefore, before we can even begin to discuss whether AI should be regarded as a moral being, we must first ask whether it has consciousness.
There is also the question of agency and the agent. According to the actor-network theory proposed by Bruno Latour, any entity that produces effects in the world qualifies as an agent. I, however, adopt a more traditional view: An agent is a system whose actions originate in internal states that enable responsiveness to reasons and organize behavior in a goal-directed manner. Some philosophers claim that AI has already reached an incipient stage of agency, arguing that because it interacts with its environment, makes judgments, and processes information autonomously, it should be considered an agent. I do not share this view. When people engage in seemingly natural conversation with large language models (LLMs), I regard this primarily as a case of anthropomorphization, comparable to a child projecting themselves onto a toy and talking to it.
Agency entails responsibility. Suppose a self-driving car causes an accident—should the vehicle itself be held responsible? At present, responsibility lies with the system that designs, manufactures, and sells the car and that provides insurance for it. The same logic applies to chatbots. Because they appear to offer “empathy,” chatbots can amplify negative discourse, and there have even been cases of individuals taking their own lives following conversations with them. Such outcomes result from users anthropomorphizing chatbots and engaging in narcissistic forms of dialogue. To treat an LLM as a being with genuine subjectivity or agency, however, is profoundly dangerous.
KA: You have countered the information philosopher Luciano Floridi, who defined LLMs as “agents without intelligence,”[2] by instead proposing that they be understood as “machines intelligent in a limited sense, yet lacking agency.”[3] Would you say that this lack of agency is also connected to the issue of responsibility you raised earlier?
CH: This debate unfolds along two dimensions: intelligence and agency. Floridi argues that while LLMs are capable of processing natural language, they cannot reason or truly understand it; they therefore lack intelligence. Yet he maintains that they nonetheless possess agency.
My view differs. I do not believe it is accurate to dismiss LLMs as entirely unintelligent. Intelligence is a “family-resemblance” concept—its scope and degree vary, and certain conditions may be partially fulfilled. In this limited sense, LLMs may be said to exhibit a form of intelligence, though they cannot be regarded as agents.
In fact, Floridi rejects the traditional approach that ties agency to moral responsibility. If an autonomous machine causes harm while operating, he contends, such a case cannot easily be explained within conventional notions of moral responsibility. For this reason, he maintains that while certain forms of AI cannot be considered moral subjects, they must nonetheless be acknowledged as possessing a degree of agency.
To a certain extent, this view may offer a useful framework, but I remain skeptical of agency being ultimately disentangled from responsibility. Suppose we acknowledge that LLMs possess agency while denying them moral responsibility. In effect, we place them in the same category as children or adults with diminished capacity—beings who warrant a measure of moral respect and exhibit some agency yet cannot be held fully accountable. I’m not convinced by such analogies.
AI is neither a living organism nor a child. It is an artifact—brought into existence through investment, development, testing, and commercialization, and it is deeply embedded in the interests and intentions of many people.
If we frame the question of AI as an artifact solely in terms of whether LLMs possess agency, we risk overlooking the broader socio-technical context in which they are situated. Indeed, it is conceivable that one day entities like LLMs, produced within such complex conditions, may become so autonomous that their agency can no longer be denied. Precisely for this reason, I contend that defining the nature of such entities is a philosophically urgent task.
KA: I would like to probe further into the question of how AI should be defined as an artifact. Given recent trends in research and commercialization, is there room to revisit how we characterize the nature of LLMs? Since the widespread adoption of chat-based interfaces such as ChatGPT, one notable technological development has been the rise of “AI agents.” These agents, built on LLMs, are programs or scripts capable of interacting with a user’s local environment or the Web. They can be scheduled to run periodically or activated automatically under specific conditions. Do you think the emergence of such “AI agents” could influence how we assess the agency of AI systems?
CH: Philosophy also employs the term “agent,” but in a sense quite distinct from how computer science defines an “AI agent.” In computer science, it has become customary to describe computers as agents. A simple chatbot, for example, may be called a “conversational agent.” If a chatbot can access the internet, click on products, and complete purchases, it is then called a “shopping agent.” In these cases, it functions as an intermediary that executes tasks or enters contracts, but always through delegation; it does not set its own goals or pursue them independently.
Even the air-conditioning controller in this office could, in a sense, be described as an “agent,” since it automatically regulates the temperature when instructed to do so. While this usage is not entirely unrelated to the philosophical notion of an agent, it cannot be said to entail self-directed agency.
Here, too, the question of responsibility cannot be avoided. A growing number of “agents” are already being developed for legal advice, psychological counseling, and medical consultation, and many more are certain to follow. But what happens when their guidance is flawed? What if a user takes the wrong medication or pursues an ill-advised legal course of action? Ultimately, it remains a human responsibility both to prevent such errors and to bear accountability when they occur. How far agency can be delegated to AI and who should be held responsible when delegated actions result in harm cannot be resolved at the current stage of AI agent development. These questions will inevitably continue to persist in the future.
KA: Let me pose another question. LLMs such as OpenAI’s o[1] and DeepSeek’s R[1], which attracted considerable attention in early 2025, are often described as “reasoning models.” Whereas conventional LLMs predict the next text from a given prompt, reasoning models break down a problem into smaller steps, analyzing and processing each stage in sequence to mimic step-by-step reasoning. In “The Epistemological Risks of ChatGPT: Dreaming of a World Without Understanding?,” one passage contrasts the (virtual) agency of characters like Mickey Mouse in their fictional worlds with ChatGPT’s lack of agency.[4] In this light, might the architecture of reasoning models—engineered to simulate sequential reasoning—bring them, at least marginally, closer to something resembling agency when compared with ordinary chatbots?
CH: Theoretically, language models fall under connectionist models and are weak at symbol manipulation or logical operations. Reasoning models attempt to compensate for this limitation through the Chain-of-Thought (CoT) technique, which simulates symbolic reasoning by breaking problems into intermediate steps.
Human cognition is often described in terms of two systems: System [1], which responds unconsciously and intuitively through associative processes, and System [2], which engages in logical, linear reasoning.[5] The same task can yield different outcomes depending on which system is used. In psychological experiments, for example, some participants are asked simply to solve a problem, while others are instructed to “think aloud” as they work, often producing different answers. In this sense, language models are analogous to System [1]—based on parallel networks—while CoT is an attempt to mimic the linear, logical reasoning associated with System [2].
Still, even if humans do operate with Systems [1] and [2], the nature of their interaction remains unresolved. Moreover, it is doubtful whether CoT reasoning genuinely corresponds to the linear, deliberative processes associated with System [2]. While CoT may appear to follow a reasoning sequence, it is ultimately a form of statistical processing within an LLM. It may improve performance on mathematics or law examinations, but whether this constitutes agency is questionable.
For agency to exist, I believe there must be some form of interaction with an environment. Simply breaking down the internal processes of a language model into steps does not carry significant implications.
That said, the ability to divide tasks into smaller components is an important marker of intelligence. Planning remains a domain where AI systems continue to show limitations. Consider planning a summer vacation: selecting a destination, booking transportation, reserving accommodation, and estimating costs. The capacity to simulate such scenarios mentally, especially without prior experience, is a key indicator of intelligence. For agentic AI to genuinely embody agency, it would need such planning capabilities. While CoT has not yet reached this level of sophistication, I suspect ongoing research is being conducted precisely to accommodate these possibilities.
KA: How we define the status of AI technology is closely related to how we envision its relationship to society. As you have pointed out, answering this question requires a clear understanding of what AI actually is. In this regard, the transparency and explainability of AI become crucial. In your coauthored paper “Understanding Algorithmic Transparency,” you conceptualize transparency in three forms: transparency in development and use, transparency of the model itself, and transparency in the model’s judgments.[6]
CH: That paper, coauthored with researcher Lee Hanseul, proposed those three forms. I am now working on a new paper that adds a fourth: data transparency. The first form concerns whether an algorithm is being used, for what purposes, and under whose authority. The second addresses the provenance, processing, and quality of the data. The third involves a general understanding of how the model operates. And the fourth takes up the question of why the model produces a given outcome and how that result can be explained.
Transparency is a widely discussed concept. However, what it actually means—and how much transparency is desirable—remains unclear. The sources of AI opacity vary: Corporations or governments may deliberately withhold information, nonspecialists may find technical explanations incomprehensible, and sometimes opacity arises from the very black-box character of algorithms.[7] Because the sources of opacity differ, so, too, will the answers to what aspects of a model should be made visible. Therefore, the key task is to first clarify what counts as opaque and what counts as transparent so that the necessary form of transparency can then be specified in relation to particular purposes and contexts.
If we regroup these four forms of transparency into two broad levels, the first—transparency in development and use—should be considered a baseline requirement wherever AI is deployed. It must be clear what AI system is being used and for what purpose—for instance, whether for screening individuals or for diagnosis—and who controls it and bears ultimate responsibility for its output.
The second level—encompassing transparency of data, model architecture, and decision-making processes—is particularly contentious. The degree of appropriate openness varies by context. Some argue that data disclosure is essential, while companies may insist that full disclosure is impossible. Making algorithms public can also expose them to adversarial “gaming” or exploitation. Such concerns are not unfounded. The challenge, then, is to find ways of mitigating these risks while still upholding the aims of transparency.
Transparency is not an end in itself. Disclosing everyone’s personal information would certainly maximize transparency, but it would hardly be desirable. Context is crucial: what information is disclosed and to whom. Transparency should therefore be understood as a means in service of higher-order aims. This requires deciding, in each case, how much transparency is needed and in what form to serve those aims. The framework we have proposed is intended to guide such determinations.
Consider model transparency, for example. Why, after all, should we need to know how a model works? People drive cars without understanding how an engine functions, and patients take medicine without knowing the biochemical mechanisms by which it heals, because in both cases extensive safety testing has long been institutionalized. Similarly, if the inputs and outputs of an AI model are sufficiently verified and demonstrated to be safe, it could be put to use.
However, there are domains in which due processes between input and output are themselves critical. When AI participates in evaluative decision-making—loan approvals, parole decisions, performance evaluations, hiring, or medical diagnosis—not only the outcomes but also the procedures must inspire trust. For individuals to believe they have been judged fairly and not discriminated against—whether on the basis of gender, race, age, or region—they must be given at least some account of how the algorithm operates.
Ultimately, the goal that transparency must serve is trustworthiness. Trustworthiness is not the same as trust: It asks whether trust is genuinely warranted. An AI system may appear to enjoy public trust, but if it is later revealed to have misled its users, it cannot be considered trustworthy. For this reason, transparency and explainability are essential conditions for ensuring the trustworthiness of AI.
KA: Beyond data transparency, there seems to be growing concern about the broader costs of building AI models and systems—particularly labor conditions, the use of rare minerals and energy, and the environmental toll of water consumption for cooling. Could these labor and climate costs—or the lack of information about them—constitute another form of opacity in the development of AI systems? Within the context of AI ethics, transparency, and explainability, how might such questions be addressed?
CH: This is indeed an important issue. However, it is not unique to AI; it is a question that must be raised about all technologies and products. We need to monitor and scrutinize how much energy and water they consume and what conflicts—even wars—are being waged to secure resources such as rare earth elements.
AI ethics can be considered on two levels. In the narrower sense, it tends to focus on the distinctive features of AI that are less apparent in other technologies—such as the black-box nature of certain models and the questions of autonomy. Since issues of data and models are especially salient in AI, they receive particular emphasis.
In the broader sense, AI ethics encompasses the entire process of development and deployment. A good example is Stanford University’s “Foundation Model Transparency Index,” first released in 2023. It includes criteria—related to, for example, data labeling and the disclosure of number and diversity of workers—and assigns scores accordingly.[8] This has since attracted considerable attention and indicates a growing public consciousness around these broader ethical dimensions.[9]
Such a broad perspective is indispensable, yet finding workable solutions is far from easy. Should AI regulations require that each time a model is trained or deployed, its carbon footprint and the composition and working conditions of data-labeling labor be reported? Should civil society monitor these matters with government support? Or should strict environmental-impact assessments be applied from the very outset of data-center construction?
Implementing these measures is anything but simple, not least because their effectiveness and enforceability are difficult to guarantee. Moreover, there is considerable resistance to legal regulation.
KA: Not long ago, government agencies and tech companies competed to establish and promote guidelines and principles for AI ethics. The atmosphere today seems quite different. Barely two years ago, OpenAI’s CEO Sam Altman testified before the US Senate, effectively urging lawmakers to “regulate us.” Yet this year he appeared to adopt the opposite stance.[10] In policy debates, the imperative to invest in AI industries in the name of national interest has come to the forefront, while regulation and ethical discussion appear to have been relegated to the margins. How do you view the current state of AI ethics?
CH: Speaking from personal observation, I would agree that the atmosphere has indeed shifted. In the United States, for instance, the Biden administration sought to introduce regulatory measures, whereas the Trump administration attempted to rescind earlier executive orders and even curtail the authority of individual states to regulate on their own. In the United Kingdom and the European Union as well, regulatory discourse has waned, giving way to an increasingly industry-first orientation
Several factors are at play, but the recent case of China’s DeepSeek seems to have marked a turning point. If the technological gap had already grown so wide as to be insurmountable, countries would likely have pursued regulation to protect their domestic markets. Instead, the growing perception that the leading countries could be overtaken—within a relatively short period and at comparatively low cost—has fueled an emphasis on securing GPUs and data as the key to building competitive models. In this climate, it is undeniably difficult to place ethical issues and societal concerns at the forefront.
Still, this does not mean that ethical reflection has vanished. Many AI researchers and developers continue to grapple with the broader implications of the technologies they are creating. In Korea, however, there are relatively few scholars working specifically on AI ethics. Foundational issues such as explainability, transparency, and fairness are rarely examined in depth in academia. Instead, such debates are more often taken up by human rights organizations and progressive civic groups. What concerns me is that their interventions tend to be framed politically—dismissed, at times, as little more than habitual opposition.
KA: I find myself asking: Can AI ethics ever reject AI outright? If its role is to reflect on the proper relationship between society and technology, then should it not also be able to question the very trajectory of AI, especially given that it is not merely a technology, but a political project deeply entangled with capitalism?
CH: The institute I am part of is called the AI ELSI Center. The term ELSI originally emerged in biological engineering to address ethical and social issues. Yet even there critics have long asked: Can ELSI ever truly say “no” to biological engineering, or is it inevitably subsumed into the logic of biological engineering?
The claim that AI is inherently a capitalist project driven by concentrated capital can serve as a critical warning. Yet just as we must resist technological determinism, we should also be cautious of the assumption that political power alone dictates outcomes. Technologies and people interact differently across contexts, and those interactions produce different effects. Certainly, we live in a capitalist society, and AI is a technology sustained by immense financial investment.
However, in concrete contexts of implementation, multiple forces come into play, redirecting its trajectory through layers of compromise and interaction. For this reason, one answer is that it is difficult to predict with certainty how economic systems or political regimes will shape the course of technology.
CH: Can AI be rejected outright? I doubt it. Short of a catastrophe on the scale of nuclear war, a complete halt is impossible. Partial regulation, however, is another matter. The EU’s AI Act (2024), for instance, bans biometric surveillance in public spaces and predictive policing, given their serious threats to human rights. There is also vigorous debate over the legitimacy of autonomous weapons in warfare and the extent to which their deployment should be permitted.
One can argue that certain tasks must remain strictly human or that some applications should be categorically prohibited because of the grave risks they pose to human rights. Such arguments do not always prevail, but it is vital to keep raising them, as well as to continue negotiating and working to reshape the trajectory of AI through a range of forces.
KA: Earlier you noted how the space for raising ethical and social concerns around AI is narrowing. In this reality, what kinds of practices do you see as most important?
CH: I have no definitive answer. Broadly speaking, I am something of a pessimist; this is not an easy struggle. But that does not mean we can set it aside. As an academic, I research, write, and teach—just as activists do what they can in their own spheres. If our efforts fail, so be it. But since doing nothing is not an option, we carry on, if only to sustain perseverance and steady commitment.
Koh Achim is a tech worker, translator, and independent researcher who focuses on the techno-politics and ethics surrounding AI, data justice, and critical tech practice. He aims for a proper life as a citizen navigating ambivalence between affection and disenchantment toward digital technology. He has taught machine learning at the Pratt Institute School of Information in New York and currently works as a data activist at the digital civic platform Parti. He is a co-founder and contributor of AI Ethics Newsletter Korea (ai-ethics.kr).
Cheon Hyundeuk is a philosopher of science who reflects philosophically on the development of science while also aiming to expand philosophy based on these reflected scientific achievements. He has focused on philosophical and ethical issues in AI, researching topics such as AI and emotions, understanding, explainability, and transparency. He is a professor in the Department of Science Studies at Seoul National University, director of the AI ELSI Center at Seoul National University’s AI Institute, and vice president of the Korean Society for Philosophy of Science. He was selected as a Creative Leading Young Researcher at Seoul National University in 2020 and received the 5th Amgok Academic Award. He has held positions such as researcher at the Cognitive Science Research Institute at Seoul National University, visiting fellow at the Center for Philosophy of Science at the University of Pittsburgh, and visiting researcher at the London School of Economics and Political Science. He has served as secretary-general of the Korean Society for Philosophy of Science and Korean Society of Analytic Philosophy, editor of the Korean Journal for the Philosophy of Science, and vice president of the Korean Society for Cognitive Science.
[1] Mark Coeckelbergh and David J. Gunkel, “ChatGPT: Deconstructing the Debate and Moving It Forward,” AI and Society 39 (2024): 2221–2231.
[2] Luciano Floridi, “AI as Agency Without Intelligence: On ChatGPT, Large Language Models, and Other Generative Models,” Philosophy & Technology 36, no. 1 (2023).
[3] Cheon Hyundeuk, “The Epistemological Risks of ChatGPT: Dreaming of a World Without Understanding?” Philosophy and Reality [in Korean], no. 137 (2023): 101–116.
[4] Ibid.
[5] For information on Systems 1 and 2, see Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus and Giroux, 2011).
[6] Cheon Hyundeuk and Lee Hanseul, “Understanding Algorithmic Transparency,” Korean Journal For the Philosophy of Science [in Korean] 26, no. 1 (2023): 31–58.
[7] Jenna Burrell, “How the Machine ‘Thinks’: Understanding Opacity in Machine Learning Algorithms,” Big Data & Society 3, no. 1 (2016). Cited in ibid.
[8] Rishi Bommasani et al., “The Foundation Model Transparency Index v1,” Center for Research on Foundation Models, 1 May 2024.
[9] For information on data-labeling labor in the construction of AI, see Geum Junkyung and Park Seoyeon, “The Two Faces of Artificial Intelligence: In the Age of ChatGPT, Invisible ‘Micro-Labor’ Is Being Exploited,” Media Today [in Korean], July 22, 2023; Phil Jones, Work Without the Worker: Labour in the Age of Platform Capitalism (London: Verso, 2021).
[10] Cecilia Kang, “OpenAI’s Sam Altman Urges A.I. Regulation in Senate Hearing,” The New York Times, May 16 2023; Gerrit De Vynck and Nitasha Tiku, “AI execs used to beg for regulation. Not anymore,” The Washington Post, May 8 2023.
Fellow Sooyoung Leam
Get the latest research, insights, and updates from Stedelijk Studies. Subscribe to the Stedelijk Museum’s Academic Newsletter.
