As lawyers, texts are – to a large extent – our ‘stock in trade.’ And so, like many other lawyers, I have become fascinated by the ability of services such as ChatGPT to use large language models (‘LLMs’) to generate human-like text. Much has been written on how LLMs can pass bar exams and how they might affect various areas of national law. But there’s something of a blind spot when it comes to how LLMs will affect international law, particularly in the realm of treaty interpretation. Yet treaty interpretation, with its complex interplay of language, context, and legal principles, provides a particularly intriguing case study for the potential impact of LLMs.
In my article ‘Large Language Models and the Treaty Interpretation Game’, published in the Cambridge International Law Journal, Vol. 12, No. 2, 2023, pp. 305–327, I examine the fascinating intersection of LLMs and treaty interpretation. I suggest that LLMs may offer efficiency benefits and increase the accessibility of international law. However, I am concerned by LLMs’ inherently backward-looking nature, which may reinforce existing biases and potentially stifle the evolution of international legal doctrine.
A high-stakes card game
In the article, I first present treaty interpretation as a high-stakes card game, an analogy first proposed by Andrea Bianchi. The players? States (and their lawyers and officials). Their cards? The principles of treaty interpretation. The audience? Well, that’s the judge (and other states) the players are trying to win over with their arguments. This framing allows us to consider a couple of questions. How might LLMs perform as players in this game? And what implications could their involvement have for the field of international law more broadly?
The strange case of the lunar tardigrades
To explore these questions, I conducted a limited case study using ChatGPT (specifically, ChatGPT 3.5) to interpret a specific phrase from Article IX of the Outer Space Treaty: “harmful contamination.” The context I provided was the 2019 incident where a failed lunar lander potentially released tardigrades (microscopic animals) onto the Moon’s surface. The results were intriguing. ChatGPT demonstrated a reasonable grasp of the basic principles of treaty interpretation, as codified in the Vienna Convention on the Law of Treaties. More specifically, ChatGPT was able to identify relevant interpretative principles, apply these principles to the specific context, and construct logical arguments in favour of a particular interpretation
While ChatGPT’s performance was far from flawless, it was certainly on par with what could be expected from a junior lawyer or law student tackling the same task. So, based on this limited case study, I tentatively suggest that LLMs such as ChatGPT can be (spoiler alert) competent players of the treaty interpretation game.
Potential benefits
Given the ready availability of LLMs, and their apparent competence in the field, might LLMs facilitate accessibility and broader engagement with international law? Perhaps. The integration of LLMs into the treaty interpretation process could offer several advantages:
1. Efficiency: LLMs can quickly process vast amounts of textual data, including travaux préparatoires, potentially streamlining the research phase of treaty interpretation.
2. Consistency: LLMs could help identify patterns and precedents across different treaties and interpretations, promoting more consistent approaches – a potential remedy to concerns regarding the fragmentation of international law.
3. Accessibility: By lowering the ‘barriers to entry,’ LLMs might allow a broader range of international actors to present sophisticated treaty interpretation arguments.
4. Multilingualism: LLMs can generally work across multiple languages, which is particularly valuable in the multilingual field of international law.
Structuring the ‘meta-game’
However, for me, the most intriguing aspect of this research was not the potential benefits of using LLMs in treaty interpretation, but rather a consideration of how LLMs might influence the ‘meta-game’ of treaty interpretation. This meta-game involves ongoing debates and reflections about the nature of treaty interpretation itself – how it should be conducted, which principles should take precedence, and how the practice evolves over time. Traditionally, this meta-game has been played primarily by experienced scholars, judges, and practitioners. Their reflections and ongoing debates shape the future of treaty interpretation. But what happens when we introduce LLMs as players? After all, an LLM’s training dataset determines how it plays the treaty interpretation game – and LLMs are trained on vast amounts of historical data, using patterns and probabilities from that data to generate responses. This means that LLMs are inherently backward-looking: their output reflects past conventions, norms, and biases. In our rush to embrace this new technology, might we inadvertently reinforce the doctrinal and structural status quo, and thereby put the brakes on the ever-evolving nature of international law?
More questions than answers
Overall, in this article I attempt to consider the broader implications of this technological shift that we are currently living through. While these tools are unlikely to replace human experts anytime soon, the potential for LLMs to enhance our interpretative capabilities is clear. Yet our enthusiasm must be tempered with caution: the integration of LLMs into treaty interpretation practice may be a double-edged sword. On one hand, LLMs have potential with respect to efficiency, consistency, accessibility, and multilingualism. On the other, LLMs raise profound questions about the nature of legal reasoning, the place of human judgment, and the future evolution of international legal doctrine. There are certainly more questions than answers in this field. How might the introduction of LLMs as players in the treaty interpretation game alter the dynamics of international legal scholarship, judicial decision-making, and diplomatic negotiations? Will LLMs democratize access to sophisticated legal argumentation, or merely reinforce the digital divide between tech-savvy international actors and those with limited (or no) access to these tools? While clear responses may not be forthcoming for some time, these questions demand attention and careful consideration.
Keywords: Artificial intelligence, large language models, treaty interpretation, data, bias
AUTHOR INFORMATION
Jack Wright Nelson is an Adjunct Research Fellow at the Centre for Banking & Finance Law, National University of Singapore.