Meta Model API

CASE STUDY

Llama-enabled data extraction and document cognition solution delivers 200% ROI at scale

At a glance

Industry

Financial services

Use case

Enabling chat-based college and course research for students

Goal

Replace expensive, time-consuming manual data capture with intelligent data extraction that can scale

Llama version

Llama 3.1 8B

Deployment

Hosted on Databricks

Download full case study

THEIR STORY

Providing clean data and critical analytics for the real estate industry

First American Data Tree LLC is a leading source of residential property data and analytics serving markets throughout the United States. They provide sought-after SaaS solutions for property data, fraud and loan quality, home equity lending, compliance, home valuation, taxes and customer acquisition. The company maintains a growing repository of over eight billion document images and holds 20 patents for data extraction technologies.

THEIR GOAL

Automate data extraction from millions of escrow and title documents

First American Data Tree LLC images and extracts data from real estate documents like deeds, mortgages, foreclosures and tax records collected from counties across the United States. Document quality ranges from pristine records made last week to 100-year-old documents with handwritten notes, stamps and watermarks.Processing their vast array of documents was manual time-consuming work. The First American team estimated that processing 300,000 documents took 30 full-time employees two to three months. They hoped to replace that expensive, error-prone process with an AI solution capable of identifying and extracting pertinent information automatically from hundreds of documents per minute.

THEIR SOLUTION

Llama-enabled data extraction agents with reasoning capabilities

First American Data Tree LLC fine-tuned Llama 3.1 8B on Databricks with their proprietary data to develop a multi-stage AI agent solution deployed on the Databricks Mosaic AI platform. The solution maintains an average F1 (pseudo accuracy) score of 90% on critical data fields while processing more than one million documents a day.

THEIR APPROACH

Fine-tuning helps Llama develop solid reasoning skills

The team fine-tuned Llama 3.1 8B on Databricks using labeled title policy data, starting with a sample of 1,000 policies and then expanding to 5,000 as they scaled up. The team trained the model on specific metadata such as policy type, easements and covenants, plus conditions and restrictions (CCRs). After training, the Llama model could identify and classify defined fields and detect related details with an average F1 (pseudo accuracy) score of over 94% on non-critical fields and over 90% on critical fields.The smaller-parameter, open-source Llama 3.1 8B was also far more cost-effective than competing commercial LLMs, which required a much larger GPU footprint and racked up per-token charges for each input and output. The team estimated running on closed commercial LLMs would be 3-4x more costly even after deep discounts. Serving the fine-tuned Llama models on Databricks further reduced costs by automatically optimizing the number of GPUs required to meet their SLA and auto-scaling as needed to keep costs in check.

THEIR SUCCESS

An AI pipeline for document processing that delivers 200% ROI

After fine-tuning and prototyping, developers scoped the production deployment to run one million documents daily — with each document averaging five pages. To handle the workload, the team stood up a cluster of 40 NVIDIA A100 GPUs hosted on the Databricks Data Intelligence Platform.The business impact of automated data extraction running at 695 documents per minute has been staggering. What used to take 30 full-time employees two to three months takes less than eight hours with the Llama solution, delivering a 200% ROI.

“After fine-tuning, Llama outperforms all the current open-source and commercial models for our use case for accuracy, cost to deploy, ease of implementation, advanced tokenization capabilities and scalability.”

Arun Rangarajan, Senior Engineering ML Manager, First American Data Tree LLC

“Llama enabled us to achieve four important document tasks in a single pipeline. By applying multiple ML techniques, the solution is able to recognize multiple occurrences of similar entities and relationships in a document, process them accurately, and prepare them for downstream use.”

Madhu Kolli, Senior Director of Data Science, First American Data Tree LLC

Previous slide

Next slide

Slide 1 of 4

Please confirm your identity