A Brief Guide to Legal AI Research

← All HX's Notes

A Mandarin Chinese translation of this article is available here.

After spending a few afternoons suggesting reading to research students I work with who want to quickly get into the field of Legal AI, I thought it might make sense to compile a guide to resources that I would recommend to newcomers to Legal AI. This article reflects my experience and interests, so the selection leans heavily towards NLP and machine learning. I mostly work in legal AI in the UK and Chinese legal contexts, so you would notice that the suggested resources for these contexts are covered in more detail than those for others. I am in the process of incorporating more resources for other jurisdictions, but as this article contains a lot of information/technical pointers that transfer across jurisdictions or are jurisdiction-agnostic, I believe it should already be useful in its present form for Legal AI researchers no matter which legal context they are interested in. It is compiled as a starting point, rather than a comprehensive survey.

I have organised the guide around three questions: What technical background is useful? What legal datasets and resources are already there? And what legal models and agents have researchers already built?

1. Start with defining the task

At the start of a Legal AI project, I find it useful to define the task’s scope precisely. I would specify the jurisdiction, area of law (e.g., criminal, civil, or administrative law), and the available sources of knowledge and data. I would also ask what the task looks like in practice; it is helpful to set out an example of the expected input and output. I would also consider how the output should be evaluated, i.e., the evaluation protocol. For computational modelling and applications, it is almost always useful to aim for something more precise than vaguely “good or bad”: something that can be evaluated, preferably quantitatively.

A useful entry point is Zhong et al.’s How Does NLP Benefit Legal System? (2020), which maps legal tasks onto NLP methods.

2. The AI and NLP background

This section may look a little overwhelming for non-tech researchers. Please be assured that you do not need to read all of these papers before starting a project. The aim of this section is to provide an inventory of basic AI/NLP concepts that you may encounter when reading Legal AI papers.

From Transformers to general-purpose language models

• Attention Is All You Need (Vaswani et al., 2017) introduces the Transformer architecture. It is the natural starting point for understanding attention, positional embeddings, and encoder and decoder components.
• Read BERT (Devlin et al., 2018) next to understand bidirectional pretrained representations and task-specific fine-tuning. Many legal classification and extraction systems were built around this paradigm, and some of them remain useful baselines.
• T5 (Raffel et al., 2019) provides a unified text-to-text view of NLP tasks and a systematic study of transfer-learning choices. You could also get a glimpse of the shape of “general-purpose” language models from this paper.
• The GPT families: GPT-1 (Radford et al., 2018), GPT-2 (Radford et al., 2019), and GPT-3 (Brown et al., 2020). They help explain the transition from generative pretraining to the ChatGPT model that is very well-known nowadays.
• Scaling law: Scaling Laws for Neural Language Models (Kaplan et al., 2020) and Training Compute-Optimal Large Language Models (Hoffmann et al., 2022). These papers present empirical studies of the relationships between model size, training data, compute, and language model performance and have been considered the main driving force behind model scaling work.

In-context learning and chain-of-thought prompting

• GPT-3 is a central reference for in-context learning: supplying examples in the prompt without updating the model’s parameters.
• The classic Jason Wei paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022) introduces what is now very well-known as Chain-of-Thought (CoT) prompting.

These are related ideas, but they should not be collapsed into one technique. A prompt can contain “end-to-end” examples without reasoning traces, or ask the model to “think step by step” without providing any examples. For legal research, I would also distinguish a model-generated explanation (“models’ reasoning”) from independently verified reasoning; model-generated intermediate “reasoning” steps still need to be checked to establish their validity and consistency with the relevant facts and legal rules.

Instruction tuning, preference learning, and reinforcement learning

• Finetuned Language Models Are Zero-Shot Learners (Wei et al., 2021) (the FLAN paper) is a useful introduction to instruction tuning. Supervised fine-tuning learns from input-output demonstrations; the data tell the model what responses to imitate.
• Training Language Models to Follow Instructions with Human Feedback (Ouyang et al., 2022) (the InstructGPT paper) covers supervised fine-tuning, learning a reward model from preferences, and reinforcement learning using that reward.

If you are particularly interested in reinforcement learning, I would follow this sequence:

1. Proximal Policy Optimization Algorithms (Schulman et al., 2017): the policy optimisation method used in many early RLHF pipelines.
2. Learning to Summarize from Human Feedback (Stiennon et al., 2020): an example of learning preferences and optimising language generation against them.
3. Direct Preference Optimization (Rafailov et al., 2023): an alternative that trains directly on preference pairs, without the separate reward model training and online RL loop of the standard RLHF pipeline.
4. Let’s Verify Step by Step (Lightman et al., 2023): introduces an important distinction between supervising a final outcome and supervising intermediate steps.
5. DeepSeekMath (Shao et al., 2024): introduces Group Relative Policy Optimization (GRPO) in mathematical reasoning.
6. DeepSeek-R1 (DeepSeek-AI, 2025): useful for understanding reasoning-oriented RL and its relationship with cold-start data and distillation.

Most of these experiments concern maths or coding. Adapting reward modelling and reinforcement learning to law requires defining reward signals that capture legal correctness and reasoning quality, including cases where multiple conclusions may be defensible, which is harder than one would think based on our team’s recent empirical work (LexReward; will be released soon).

Retrieval-augmented generation (RAG)

Law is a knowledge-heavy field, which naturally makes retrieval-augmented generation (RAG) a useful design for legal AI tasks. You can read more about RAG here: Retrieval-Augmented Generation (Lewis et al., 2020). When using RAG in legal applications, there are a few questions worth considering/checking: Which sources should be retrieved? Did the model use them correctly? If a search failed, did the system recognise the failure or continue with an unsupported answer?

Agents (Note: This section will be expanded further)

• Plan-and-Solve (Wang et al., 2023) prompts a language model to first devise a plan that decomposes a problem into smaller subtasks, then solve them step by step.
• Plan-and-Execute (Topsakal and Akinci, 2023) separates high-level planning from execution, using a planner to outline the steps and a tool-using agent to carry them out.
• ReAct (Yao et al., 2023) interleaves reasoning and actions, allowing observations from an environment to inform subsequent steps.

3. Datasets and benchmarks

I roughly group resources by jurisdiction below, with separate space for multilingual and cross-jurisdiction work. Note that language and jurisdiction are different attributes; for example, English does not imply US/UK law. Currently, the compilation covers textual data only; I will probably add some multimodal datasets in future as there has been some new work in this direction.

Before we get to the resource lists, it also helps to distinguish what each resource provides:

Europe and multilingual resources

• LexGLUE (Chalkidis et al., 2022) is a good starting benchmark. It provides seven English-language tasks covering ECtHR, US Supreme Court, EU legislation, contract, and case-holding datasets.
• MultiEURLEX (Chalkidis et al., 2021) provides EU legislation with EuroVoc labels across 23 languages.
• LEXTREME (Niklaus et al., 2023) provides 11 datasets covering 24 languages and several legal NLP tasks.
• MultiLegalPile (Niklaus et al., 2024) is a large multilingual legal text corpus, spanning 24 languages and 17 jurisdictions.

China

• CAIL2018 (Xiao et al., 2018) is a large criminal case dataset for tasks including charge prediction, applicable law prediction, and sentencing-term prediction. It establishes several influential task formulations.
• JEC-QA (Zhong et al., 2020) contains bar exam questions. It is useful for studying legal question answering.
• LeCaRDv2 investigates Chinese criminal case retrieval, with expert annotations provided.
• LexChain (Xie et al., 2026) structures Chinese tort case analysis into legal element identification, liability analysis, and judgment synthesis. It is useful for investigating where errors arise along an explicit analysis process.
• LexRel (Cai et al., 2026) studies legal relation extraction in Chinese civil cases using a structured taxonomy. It directs attention to the parties and relations that a system must represent before producing a useful analysis.
• LexIssue (Xie et al., 2026) addresses issue identification in Chinese civil litigation through generation and hierarchical classification.
• For broader LLM evaluation, LexEval (Li et al., 2024), LawBench (Fei et al., 2024), and PLawBench (Shi et al., 2026) are worth having a look at. They all offer collections of legal evaluation tasks and are good starting points if you are looking for Chinese legal benchmarks to evaluate your models.

US

• Pile of Law (Henderson et al., 2022) is a large English-language legal and administrative text corpus.
• LegalBench (Guha et al., 2023) offers 162 tasks designed around different forms of legal reasoning.
• CUAD (Hendrycks et al., 2021): expert-annotated contract review data, useful for identifying legally salient portions of contracts.
• ContractNLI (Koreeda and Manning, 2021): document-level inference over contracts, with entailment, contradiction, and supporting evidence spans.
• LegalBench-RAG (Pipitone and Alami, 2024): fine-grained retrieval evaluation built from existing legal datasets, useful for asking whether a retriever locates the passages needed to answer a question.

UK (Note: This section will be expanded further)

• Cambridge Law Corpus (Östling et al., 2023) contains over 250,000 UK court cases and an expert-annotated subset for outcome extraction.

Germany

• GerDaLIR (Wrzalik and Krechel, 2021) provides a German legal retrieval dataset built from court decisions.
• GerLayQA (Büttner and Habernal, 2024) connects laypeople’s legal questions, lawyers’ answers, and references to German legal provisions. It is especially interesting for studying the gap between everyday descriptions of a problem and formal legal language.

India

• ILDC (Malik et al., 2021) provides Indian Supreme Court cases for decision prediction, with expert explanation annotations for a subset.
• LegalEval (Modi et al., 2023) covers rhetorical roles, named entities, and judgment prediction with explanation. Identifying what a paragraph does—describing facts, reporting an argument, or giving a reason—can be valuable before attempting an end-to-end analysis.
• IL-TUR (Joshi et al., 2024) investigates Indian legal understanding tasks, including retrieval, summarisation, translation, and judgment-related tasks.

Japan

• COLIEE (Goebel et al., 2026) provides Japanese civil law materials and bar exam questions.

4. Legal models and systems

• LEGAL-BERT (Chalkidis et al., 2020) is an influential reference for domain-adapted Transformer-based models.
• Lawformer (Xiao et al., 2021) adapts a Longformer-based architecture to Chinese legal documents, emphasising long document handling.
• For generative models, DISC-LawLLM (Yue et al., 2023) combines Chinese legal instruction tuning with legal knowledge retrieval.
• SaulLM-7B (Colombo et al., 2024) trains an English-language legal model based on Mistral, with legal-domain training and instruction adaptation.
• LegalOne (Li et al., 2026) is a more recent model combining legal mid-training, supervised training with distilled reasoning data, and curriculum-based RL.

5. Reward modelling and reinforcement learning in law

This is the part of the field that currently interests me most. Once we can generate many plausible answers or arguments, the question becomes: what feedback (reward) should we provide to the models to make the next answer better?

• Building Legal Reward Models for Grounding and Abstention (Franzone et al., 2026): this recent paper introduces LegalRewardBench and investigates contextual reward modelling, i.e., assessing answers against supplied evidence. The study constructs preferences from generated responses and automated annotations, with data primarily grounded in Victorian criminal law and additional US evaluations.
• LegalΔ (Dai et al., 2025) combines distilled reasoning data with RL and a reward based partly on the contribution of reasoning to answer prediction.

6. Legal agents

An agentic system makes sequential choices: what to search, which document to inspect, what tool to call, etc. This adds planning and state management compared with “static” LLMs.

Benchmarks and environments

• LegalAgentBench (Li et al., 2025) evaluates agents using Chinese legal resources and tools, including tasks requiring multiple steps. Its repository is a useful entry point for examining the actual environment.
• Ready Jurist One (Jia et al., 2025) introduces J1-ENVS, with six scenarios spanning legal consultation, document drafting, and judicial adjudication in Chinese legal settings. Its companion framework, J1-EVAL, evaluates both final outcomes and procedural compliance. I would read it alongside LegalAgentBench to study what changes when evaluation includes information gathering and role interaction.
• Harvey’s Legal Agent Benchmark (LAB) (Harvey AI, 2026): an open-source benchmark built to evaluate and improve agent capabilities for supporting legal work. Each task consists of an instruction, a client matter containing relevant materials, and a requirement that the agent produce a work product for review, mirroring how work is performed and reviewed at law firms.
• LegalWorld (Zuo et al., 2026) looks at longer workflows across stages of Chinese civil litigation, with persistent case information and a long-horizon evaluation setting.

In agentic settings, evaluations usually assess task success alongside tool-call correctness, evidence quality, intermediate steps, budget use, etc.

Single-agent systems (TODO)

Multi-agent systems (MAS)

Legal settings provide natural roles for multiple agents. Below are a few recent MAS papers:

• AgentCourt (Chen et al., 2025) uses courtroom simulation and adversarial interaction, with stored experience and legal materials.
• LinguaGame (Ye et al., 2026) frames language interaction through signalling games and uses a training-free approach to strategic inference. It is relevant to debate and courtroom simulation.
• GARL (Ye et al., 2026) explicitly studies game-theoretic multi-agent reinforcement learning. Its legal application concerns ranking legal issues in dispute through strategic allocation and arbitration.

7. Further reading

Lilian Weng’s blog is one of my favourite places to build a conceptual map before following references into individual papers. For this article’s themes, I would start with LLM Powered Autonomous Agents and Reward Hacking in Reinforcement Learning.

I also recommend following Bingxiang He’s research, particularly the work on on-policy distillation (OPD). Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe examines when distillation succeeds and how teacher–student compatibility affects it. Rethinking On-Policy Distillation of Large Language Models II: One Training Example examines training data requirements and the states visited during student rollouts.