FRONTEO is a global eDiscovery leader and a pioneer in AI Assisted eDiscovery (AI Review). FRONTEO developed KIBIT, an AI engine, and placed it at the heart of our analytics review tool, KIBIT Automator (KAM), to deliver the advantages of AI Review to clients. KIBIT is a natural language processor that requires only a small amount of training data to maximize review efficiency.
FRONTEO has been using KIBIT for several years not only within e-Discovery review workflows in cross-border cases, but also within Japan in industry sectors from finance to government to help cut through voluminous data to find the information required to run businesses more efficiently.
FRONTEO developed an AI Review workflow that maintains review quality while increasing overall review speed. KIBIT compares review target documents against responsive and non-responsive training documents, assigns to each document a Relevancy Score between 0 and 10,000, and then reorganizes and groups the review target documents according to the Relevancy Score. With KIBIT, we can create a defensible cut-off point based on the Relevancy Scoring, thereby greatly reducing the number of documents requiring manual review. The manual review portion is more effective because KIBIT prioritizes probable responsive documents by score.
In addition to KIBIT’s AI-powered scoring of documents, KAM offers a suite of reporting functions and analytics that makes the review process transparent and defensible, and supports review quality control. We estimate that, due to the impact of the AI Review workflow, AI review is nearly twice faster than standard linear review.
FRONTEO’s AI Review workflow harnesses the power of AI throughout the review workflow to deliver transparency, defensibility, and efficiency – with built-in validation checks. AI Review is the next step in the Technology Assisted Review (TAR) arena, which is already widely accepted in eDiscovery. Many review platforms offer some kind of TAR, and some US government agencies use TAR and actively encourage its use.
FRONTEO calls its unique combination of technology and AI-driven workflow KIBIT Automator, or KAM. FRONTEO aims to make every stage of the AI Review process more efficient and effective through AI.
Figure 1: AI Review Process
Figure 1 above illustrates the AI Review workflow. First, the review target is established. The entire document corpus can be designated for review, or the review target can be narrowed. This narrowing is frequently accomplished through a keyword search, but in some cases, TAR (such as Concept Analysis) is used to narrow the review target.
Next, the AI engine, KIBIT, is trained. FRONTEO samples the review target and a small amount is reviewed and coded to make the KIBIT training data set. After KIBIT is trained with the training data, KIBIT’s training model is evaluated by typical statistical indicators such as Recall, Precision, and Elusion rate. Once the client/law firm agrees that KIBIT’s statistical indicators are acceptable, the KIBIT training model is applied to the entire review target.
KIBIT scores each document based on an assessment of how likely the document is responsive on a scale of 10,000 down to 0 (Relevancy Score). Relevancy Scoring can be used for two things. First, documents can be batched to reviewers by KIBIT based on the similarity of relevancy scoring – meaning similar documents can be reviewed together. The difference between KIBIT and a typical similarity search is that KIBIT calculates the relevance score by analyzing the context and value of the document, not the superficial structure (similar sentences, words, etc.). Second, a “cut off” can be established whereby documents below a certain score will be deemed non-responsive, and not be reviewed. The cut-off point can be determined on a case-by-case basis, based on the statistical model associated with that case. Review resources are then focused on documents most likely to contain responsive documents.
To confirm that the cut-off point selected is appropriate, an Elusion test is performed on a sample of the cut-off documents to confirm these are non-responsive. A statistical model is used to help assess whether the frequency of any responsive documents found below the cut off is reasonable.
FRONTEO also offers a truly unique AI-driven tool within its AI Review for quality control (QC) of the documents subject to manual review: a “QC Heat Map.” In this AI-driven feature, the results of human reviewers are measured against the predicted classification of the KIBIT score, and the results of the QC can be used to evaluate the efficacy and quality of the human reviewer. Thus, the QC Heat Map and the use of feedback to improve batching to reviewers brings AI capability into every stage of the AI Review process.
Compared to linear review, AI Review reduces the number of documents that must be reviewed by humans, increases review speed by reorganizing documents in order of responsiveness and by assigning batches based on relevancy scoring, and improves review quality through implementing AI within the QC process.
At FRONTEO, we know from experience that review speed increases when reviewers review similar documents together. Thus, for AI Review, we use KIBIT’s Relevancy Score to create high scoring batches and low scoring batches.
Figure 2: Assign batches to reviewers based on KIBIT’s Score