Meta Global AI Developer Hackathon. $1M in prizes.Get notified
LLAMA 3 8B INSTRUCT
Case-based Research
97%+
accuracy on the easiest 98% of research cases90%+
accuracy on the most difficult 2% of research cases5x
less expensive than the leading 4th generation LLM in tests30x
faster response times in production versus the leading 4th generation LLM tests*All results are self-reported and not identifiably repeatable. Generally expected individual results will differ.
CASE STUDY
Uncovering the truth in the most challenging research cases
At a glance
Industry
TechnologyUse case
Automating document processingGoal
Classify and assess the most complex information quickly and accuratelyLlama version
Llama 3 8B Instruct (fine-tuned)Deployment
PredibaseTHEIR STORY
Helping clients extract objective analyses from complex documents and records
Case-Based Research (CBR) is a pseudonym for a successful SaaS company that uses AI and machine learning (ML) to extract salient details from multiple document types and create objective research reports.
THEIR GOAL
Automate the most difficult research cases with Generative AI
Research is a challenging, time-consuming task for human experts. A thorough manual review of a single research case can take hours of an expert’s time.The CBR platform automates research services for millions of cases a month. A machine learning model handled 98% of research cases, but the remaining 2% were too complex to process, leaving clients with tens of thousands of cases to review by hand.Even expert professionals struggled to review these outlying cases, which could have more than 230 distinct data categories and noisy documentation. The team hoped large language models could do the job within CBR’s budget and SLAs.
THEIR SOLUTION
Fine-tuned Llama cracked the hardest research cases
CBR repowered its platform with a fine-tuned Llama 3 8B Instruct model. The model reached 90% accuracy on the most difficult research cases, outperforming the leading 4th generation LLM in accuracy and response times. By using a lightweight, small language model (SLM), CBR met their platform’s low-latency SLAs with far less computing overhead and far lower costs than a commercial model.
During evaluation, fine-tuned Llama 2 7B delivered 9.4% higher accuracy and 30x faster response times than the leading 4th generation LLM
THEIR APPROACH
Predibase delivers fine-tuned models at scale
CBR used Predibase for training and production. The platform uses Low-Rank Adaptation (LoRA) and Parameter Efficient Fine-Tuning (PEFT) to deliver fine-tuned performance without retraining the entire model. This technique drastically reduces training time and computing overhead in production.After fine-tuning on a 150,000-row dataset, Llama 3 8B Instruct delivered 90% accuracy in production and maintained 0.15 sec response times under full-scale traffic loads.
CBR uses increasingly sophisticated classification services to process information efficiently.
THEIR SUCCESS
Llama delivers fast, objective analysis for complex research cases
With Llama 3 8B Instruct and the Predibase platform, CBR has automated its most difficult research cases, saving clients hours of expert labor and ensuring that complex research cases receive thorough, unbiased reviews.Projected results:
- 97%+ accuracy on the easiest 98% research cases
- 90%+ accuracy on the most difficult 2% of research cases
- 5x faster onboarding for semiconductor professionals
- 30x faster response times in production versus the leading 4th generation LLM tests
“Using Llama and Predibase, we've achieved lightning-fast inference and reduced costs by 5x compared to the leading 4th generation LLM. Most importantly, we’ve built a better product for our customers, leading to more objective, transparent and efficient research.”
Staff ML Engineer, CBR
“By using Predibase to fine-tune and serve a Llama 3 8B Instruct model, we achieved better accuracy, faster response times and 5x reduction in costs compared to traditional generative AI approaches utilizing commercial LLMs.”
Staff ML Engineer, CBR
“Our best-performing model was Llama 3 8B Instruct, which we fine-tuned on Predibase. We achieved an accuracy of 90% for the most challenging 2% of cases, outperforming both the leading 4th generation LLM and all of our other fine-tuning experiments. Additionally, Predibase consistently provided highly accurate results, whereas some of the other tools we tested were less reliable across fine-tuning runs.”
Staff ML Engineer, CBR
Previous slide
Next slide
Start building
Explore more
ConsumerBusiness
Shopify | Llama case studiesShopify uses Llama to generate product pages, localize content, and automate support, helping developers scale workflows and save time.Read more
ConsumerTech
Scribd, INC | Llama case studiesDelivering faster, cheaper and more accurate results with a Llama-powered AI content discovery assistant.Read more
Stay up-to-date
Our latest updates delivered to your inbox
Subscribe to our newsletter to keep up with the latest AI updates, releases and more.