Meta Global AI Developer Hackathon. $1M in prizes.Get notified
LLAMA 2 7B-BASED LLaVA
Shopify
Tens of billions
of tokens processed dailyImproved
merchant experienceMore accurate
product attributes and metadataIntelligent
customer search and discovery*All results are self-reported and not identifiably repeatable. Generally expected individual results will differ.
CASE STUDY
Optimizing Shopify listings with information extracted from product images
At a glance
Industry
eCommerceUse case
Generating product attributes with data extracted from product imagesGoal
Assist merchants and improve listing performanceLlama version
Fine-tuned LLaVA model (based on Llama 2 7B)Deployment
LMdeploy on 100 NVIDIA A100 GPUsTHEIR STORY
Making commerce better for everyone
Shopify is an all-in-one commerce platform that powers millions of businesses. By making it easier to start, run and grow a business, Shopify helps people around the world achieve independence through commerce.
THEIR GOAL
Improving listing accuracy with image-aware AI
Shopify merchants upload millions of product images to the platform every day. Each product image contains valuable information including the product’s color, material, size and more.As multi-modal models matured, Shopify developers saw a new way to help merchants create high-performance listings: use the visual information inside product images to help merchants create more effective product attributes.
THEIR SOLUTION
Dual real-time and offline agents automate product tagging and categorization
Faced with more than 10,000 possible attribute categories, millions of new products each day and billions of existing images, Shopify developers realized they needed to split the solution into two use cases.A real-time AI agent helped merchants create accurate product attributes as they entered new product listings. A second offline agent extracted attributes from the billions of images and product descriptions on the platform and updated listings for better search performance.
Open-source LLaVA delivered the right mix of capabilities and low cost
Developers began with proprietary models, but per-token charges made them impractical for processing billions of tokens per day. Llama-based Large Language and Vision Assistant (LLaVA) — an open-source multi-modal model that combines a vision encoder with Llama 2 — delivered competitive results without per-token charges, making it an ideal choice for production inference at a massive scale.THEIR APPROACH
Quantized training saved time and reduced computing costs
The Shopify team needed higher performance than stock LlaVA could deliver, so developers retrained a Llama 2 7B-based LLaVA model on a mix of real-world and synthetic data, all vetted by human evaluators.The team leveraged Quantized Low-rank Adaptation (QLoRA) for the first training rounds. QLoRA used quantization and selective parameter updating to reduce memory footprints and compute workloads so training could run on more modest hardware.Training Llama 7B at lower precision with QLoRA got predictions within 95% of the required accuracy. To reach their final target, the team switched to full-precision training on bare metal resources in the Google Cloud Platform.
Shopify retrained LLaVA to extract product attributes from images, create uniform metadata, categorize listings and provide conversational search services for customers.
THEIR SUCCESS
Happier merchants, more accurate product attributes
The Llama-powered merchant assistant made it far easier to correctly categorize product listings and create accurate attributes. The offline solution delivered platform-wide improvements as it reviewed billions of images, improved descriptions and added nuance by applying more than 10,000 possible metadata categories.The improved listings delivered major benefits to shoppers. Products surfaced accurately, so shoppers were able to find what they were looking for fast — and that increased sales sitewide.
- Tens of billions of tokens processed daily
- Improved merchant experience
- More accurate product attributes and metadata
- Intelligent customer search and discovery
“Closed models performed well, but their pricing structures made them prohibitively expensive. When you need to run billions of multi-modal inferences a day, per-token charges aren’t sustainable.”
Kshetrajna Raghavan, Machine Learning Leader, Shopify
“We were pleasantly surprised to find that with fine-tuning, the smaller Llama 7B model delivered the accuracy we needed. Using a smaller parameter size sped up training and saves compute cycles in production.”
Agatha Krajewski, Director of Product, Shopify
“Analyzing every image on the platform and generating metadata is an immense task. Using smaller-parameter Llama models makes it possible without expensive, bleeding-edge hardware.”
Kshetrajna Raghavan, Machine Learning Leader, Shopify
“Without Llama-powered, open-source LLaVA, providing real-time listing support for merchants would be prohibitively expensive, let alone optimizing Shopify's entire product catalog. Llama 2 7B's small footprint and zero per-token fees made our project both technically and economically feasible.”
Agatha Krajewski, Director of Product, Shopify
Previous slide
Next slide
Start building
Explore more
ConsumerTech
Scribd, INC | Llama case studiesDelivering faster, cheaper and more accurate results with a Llama-powered AI content discovery assistant.Read more
Stay up-to-date
Our latest updates delivered to your inbox
Subscribe to our newsletter to keep up with the latest AI updates, releases and more.