Publications

Technology-assisted review (TAR) refers to iterative workflows that combine human review with AI techniques such as active learning and LLMs to minimize both time and manual effort while maximizing effectiveness. The use of TAR in the discovery process in civil litigation is a multi-billion dollar industry which has had an enormous impact on the practice of law. This application of TAR, along with applications to internal investigations and sunshine law requests, constitutes the largest and most well-established application of AI in the law. The history of TAR rollout also has lessons for the adoption of AI technology more broadly in the law. The morning portion of the tutorial will cover key concepts in TAR, an overview of the technologies and workflow designs used, the basics of practical evaluation methods, and legal and ethical implications of TAR deployment. The afternoon will go into more technical depth on the implications of TAR workflows for supervised learning algorithm design, how generative AI is being applied in TAR, more sophisticated evaluation techniques (including for generative AI), and a wide range of open research questions.

High Recall Retrieval (HRR) tasks, including eDiscovery in the law, systematic literature reviews, and sunshine law requests focus on efficiently prioritizing relevant documents for human review.Technology-assisted review (TAR) refers to iterative human-in-the-loop workflows that combine human review with IR and AI techniques to minimize both time and manual effort while maximizing recall. This full-day tutorial provides a comprehensive introduction to TAR. The morning session presents an overview of the key technologies and workflow designs used, the basics of practical evaluation methods, and the social and ethical implications of TAR deployment. The afternoon session provides more technical depth on the implications of TAR workflows for supervised learning algorithm design, how generative AI is can be applied in TAR, more sophisticated statistical evaluation techniques, and a wide range of open research questions.

Review for responsiveness is a recall-oriented document classification task central to civil litigation. In large legal matters, it may involve the coding of millions of documents by teams of dozens to hundreds of contract attorneys. We describe a prototype document review system based on a large language model (LLM) for replacing the first level of attorney review. Our system accepts the same guidance—a written review protocol—that would be provided to a human review team. We tested our prototype in the context of a live legal matter, evaluating both human review and our LLM-based system against a gold standard coded by expert senior attorneys. Our prototype achieved an estimated 96% recall and 60% precision without matter-specific tuning, and has numerous avenues for further improvement.

Technology-assisted review (TAR) workflows are central to electronic discovery (eDiscovery). Researchers have proposed many methods for evaluating TAR workflows, but this research has had little impact on eDiscovery practice. We examine the operational constraints faced by eDiscovery reviewers and managers, and show how past evaluation proposals are inconsistent with their needs. We then present a new evaluation approach for one-phase TAR workflows based on confidence sequences. Our approach provides a review manager with complete control over the design and duration of the TAR workflow, as well as the amount and timing of review of evaluation documents. Evaluation documents can be reused for supervised learning while preserving valid frequentist confidence intervals on recall at all points during review. The method is expensive in terms of sample size but plausible for large scale reviews, and has many opportunities for improvement. Recipient of the ICAIL '23 Best Paper award.