Meta Global AI Developer Hackathon. $1M in prizes.Apply Now
LLAMA 3.1 8B
First American Data Tree LLC
695
documents processed per minute, each with 75-175 fields90%
accuracy (F1 score) for critical fields94%
accuracy (F1 score) for non-critical fields4x
lower inference cost than competing closed models200%
ROI*All results are self-reported and not identifiably repeatable. Generally expected individual results will differ.
CASE STUDY
Llama-enabled data extraction and document cognition solution delivers 200% ROI at scale
At a glance
Industry
Financial servicesUse case
Enabling chat-based college and course research for studentsGoal
Replace expensive, time-consuming manual data capture with intelligent data extraction that can scaleLlama version
Llama 3.1 8BDeployment
Hosted on DatabricksTHEIR STORY
Providing clean data and critical analytics for the real estate industry
First American Data Tree LLC is a leading source of residential property data and analytics serving markets throughout the United States. They provide sought-after SaaS solutions for property data, fraud and loan quality, home equity lending, compliance, home valuation, taxes and customer acquisition. The company maintains a growing repository of over eight billion document images and holds 20 patents for data extraction technologies.
THEIR GOAL
Automate data extraction from millions of escrow and title documents
First American Data Tree LLC images and extracts data from real estate documents like deeds, mortgages, foreclosures and tax records collected from counties across the United States. Document quality ranges from pristine records made last week to 100-year-old documents with handwritten notes, stamps and watermarks.Processing their vast array of documents was manual time-consuming work. The First American team estimated that processing 300,000 documents took 30 full-time employees two to three months. They hoped to replace that expensive, error-prone process with an AI solution capable of identifying and extracting pertinent information automatically from hundreds of documents per minute.
THEIR SOLUTION
Llama-enabled data extraction agents with reasoning capabilities
First American Data Tree LLC fine-tuned Llama 3.1 8B on Databricks with their proprietary data to develop a multi-stage AI agent solution deployed on the Databricks Mosaic AI platform. The solution maintains an average F1 (pseudo accuracy) score of 90% on critical data fields while processing more than one million documents a day.
THEIR APPROACH
Fine-tuning helps Llama develop solid reasoning skills
The team fine-tuned Llama 3.1 8B on Databricks using labeled title policy data, starting with a sample of 1,000 policies and then expanding to 5,000 as they scaled up. The team trained the model on specific metadata such as policy type, easements and covenants, plus conditions and restrictions (CCRs). After training, the Llama model could identify and classify defined fields and detect related details with an average F1 (pseudo accuracy) score of over 94% on non-critical fields and over 90% on critical fields.The smaller-parameter, open-source Llama 3.1 8B was also far more cost-effective than competing commercial LLMs, which required a much larger GPU footprint and racked up per-token charges for each input and output. The team estimated running on closed commercial LLMs would be 3-4x more costly even after deep discounts. Serving the fine-tuned Llama models on Databricks further reduced costs by automatically optimizing the number of GPUs required to meet their SLA and auto-scaling as needed to keep costs in check.
THEIR SUCCESS
An AI pipeline for document processing that delivers 200% ROI
After fine-tuning and prototyping, developers scoped the production deployment to run one million documents daily — with each document averaging five pages. To handle the workload, the team stood up a cluster of 40 NVIDIA A100 GPUs hosted on the Databricks Data Intelligence Platform.The business impact of automated data extraction running at 695 documents per minute has been staggering. What used to take 30 full-time employees two to three months takes less than eight hours with the Llama solution, delivering a 200% ROI.
- 90% accuracy (F1 score) for critical fields
- 94% accuracy (F1 score) for non-critical fields
- 4x lower inference cost than competing closed models
- 200% ROI
“After fine-tuning, Llama outperforms all the current open-source and commercial models for our use case for accuracy, cost to deploy, ease of implementation, advanced tokenization capabilities and scalability.”
Arun Rangarajan, Senior Engineering ML Manager, First American Data Tree LLC
“Llama enabled us to achieve four important document tasks in a single pipeline. By applying multiple ML techniques, the solution is able to recognize multiple occurrences of similar entities and relationships in a document, process them accurately, and prepare them for downstream use.”
Madhu Kolli, Senior Director of Data Science, First American Data Tree LLC
“The Llama model can reason with deep expertise and maintain a 90% F1 score in production. After processing over 50 million starter policies, only a few hundred thousand required human feedback.”
Arun Rangarajan, Senior Engineering ML Manager, First American Data Tree LLC
“We’re at the forefront of utilizing and experimenting with model choices. The velocity at which open-source models are being released enables us to quickly adapt to various use cases.”
Madhu Kolli, Senior Director of Data Science, First American Data Tree LLC
Previous slide
Next slide
Start building
Explore more
ConsumerBusiness
Shopify | Llama case studiesShopify uses Llama to generate product pages, localize content, and automate support, helping developers scale workflows and save time.Read more
ConsumerTech
Scribd, INC | Llama case studiesDelivering faster, cheaper and more accurate results with a Llama-powered AI content discovery assistant.Read more
Stay up-to-date
Our latest updates delivered to your inbox
Subscribe to our newsletter to keep up with the latest AI updates, releases and more.