Confidence Sequences for Evaluating One-Phase Technology-Assisted Review
Technology-assisted review (TAR) workflows are central to electronic discovery (eDiscovery). Researchers have proposed many methods for evaluating TAR workflows, but this research has had little impact on eDiscovery practice. We examine the operational constraints faced by eDiscovery reviewers and managers, and show how past evaluation proposals are inconsistent with their needs. We then present a new evaluation approach for one-phase TAR workflows based on confidence sequences. Our approach provides a review manager with complete control over the design and duration of the TAR workflow, as well as the amount and timing of review of evaluation documents. Evaluation documents can be reused for supervised learning while preserving valid frequentist confidence intervals on recall at all points during review. The method is expensive in terms of sample size but plausible for large scale reviews, and has many opportunities for improvement. Recipient of the ICAIL '23 Best Paper award.