To evaluate KAM’s effectiveness for speed of review, we simulated a standard linear review using publicly available Enron data, a review protocol, and an actual team of reviewers managed by an attorney. We utilized the results of this simulated review to estimate the increased effectiveness of the AI Review.
We simulated a linear review using data from the 2000 Enron case, actual reviewers, and a review protocol. We performed a keyword search to create a review target. The review target had 9,681 documents. Our team of six 1st level reviewers took a total of 6 days (261 hours) to review the review target of 9,681 documents. The average review speed was 37 documents/hr, and the review target had a richness of 4.8%.
We used the coded documents from the mock linear review to simulate KAM’s performance. We used 365 coded documents as training data. We found that, if we had used an AI Review process with KAM, we could have eliminated 68.9% of the review target (6,668 documents out of 9,681) with a Recall rate of 86.67%, an Elusion Rate of 2.98%, leaving only 3,013 documents to review. We determined (based on prior experience) the rates to be acceptable.
Based on historical data, we know that generally high score batches have a review rate of 20 documents/hr. while low score batches are reviewed at 60 documents/hr. We estimated the number of higher scoring (likely relevant) documents based on a 5% richness versus the number of documents to be reviewed.
Thus, we estimate that, compared to the actual linear review we conducted, if we had used KAM and the KIBIT technology, we could have completed review of the overall review target (including cut off documents) in 55 hours, at a rate of 177 documents/hr. Even excluding the cut off documents, we estimate the review rate would have be approximately 55 documents/hr, still much faster than linear review.
KAM has also been used in real cases to cut off and prioritize documents. In a very recent case with a tight deadline, the review target was approximately 30,000 documents collected from 10 custodians. Using 380 coded documents as KIBIT training data, we were able to eliminate 13,500 from the review (45% of the overall review target), with an Elusion rate of 5.5%. Thus, reviewers only reviewed 16,500 documents out of the original 30,000 documents. As a result, the overall review speed, including cut-off documents, was 80 documents/hr.
KAM has proven to be – in both mock reviews and actual matters – a game-changing way to incorporate AI into the entire review workflow. FRONTEO has established that KAM is highly effective and requires very few documents to develop a reliable training model. KIBIT creates and validates a defensible cut-off point, thereby greatly reducing the number of documents requiring manual review. Even the manual review portion is more effective due to the document prioritization batching based on relevancy scoring and the use of AI within the review QC process (QC Heat Map). We estimate that the overall review speed in an AI review is nearly twice as fast as a standard linear review. And, the QC Heat Map can support a comprehensive and more accurate QC of the linear review process. Overall, AI Review with KAM achieves dramatically increased review speeds while maintaining – or improving – review quality.
We would like to thank our FRONTEO colleagues for their help with this paper. We would also like to thank Edward H. Rippey of Covington & Burling, LLP for his feedback.
Written by :
Shinya Iguchi, Director of Product Development
Anitha Henderson, Attorney Manager
Sayaka Nishino, AI Review Manager
Akiteru Hanatani, Data Scientist and Development Leader
Hideki Takeda, Chief Technology Officer
FRONTEO’s Data scientists and Review experts are standing by to answer your questions.CONTACT US