Category
Benchmark
General
MMLU Chat
(0-shot, CoT)
MMLU PRO
(5-shot, CoT)
Instruction Following
IFEval
Code
HumanEval
(0-shot)
MBPP EvalPlus
(base) (0-shot)
Math
MATH
(0-sho, CoT)
Reasoning
GPQA Diamond
(0-shot, CoT)
Tool use
BFCL v2
(0-shot)
Long context
NIH/Multi-needle
Multilingual
Multilingual MGSM
(0-shot)
Pricing*
1M Input tokens
(Cheapest among providers)*
1M Output tokens
(Cheapest among providers)*
Llama 3.1 70B
86.0
66.4
Llama 3.3 70B
86.0
68.9
Amazon Nova
Pro
85.9
-
Llama 3.1 405B
88.6
73.4
Gemini Pro
1.5
87.1
76.1
GPT-4o
87.5
73.8
Claude 3.5
Sonnet
88.9
77.8
* API Pricing based on publicly available data on Artificial Analysis as of 12/3/24.
Category
Benchmark
General
IFEval
Math
Reasoning
Tool use
Long context
Multilingual
Llama 3.2 1B
49.3
41.6
Llama 3.2 3B
63.4
40.1
57.8
31.2
69.0
34.5
Modality
Image
Text
Category
Benchmark
College-level Problems and Mathematical Reasoning
Llama 3.2 11B
50.7
33.0
Llama 3.2 90B
60.3
45.2
Claude3 - Haiku
50.2
27.3
GPT-4o-mini
59.4
42.3
87.5
80.5
86.0
67.8
48.0
77.5
97.5
86.9
$0.1
$0.4
92.1
88.4
87.6
77.0
50.5
77.3
97.5
91.1
$0.1
$0.4
92.1
89.0
-
76.6
-
-
-
-
$0.80
$3.20
88.6
89.0
88.6
73.9
49.0
81.1
98.1
91.6
$1.0
$1.8
81.9
89.0
87.8
82.9
53.5
80.3
94.7
89.6
$1.30
$5.0
84.6
86.0
83.9
76.9
47.5
74.0
-
90.6
2.5$
10.0$
89.3
93.7
86.8
78.3
65.0
79.3
99.4
92.8
$3.0
$15.0
16.8
59.5
44.4
30.6
59.4
27.2
41.2
25.7
13.5
38.0
20.3
75.0
24.5
19.0
77.4
77.7
48.0
78.6
32.8
69.8
67.0
34.3
63.3
19.8
84.7
58.2
13.9
61.9
62.5
23.8
76.7
27.5
61.1
27.4
21.0
-
-
-
40.2
12.8
59.2
86.2
44.2
87.4
31.9
81.4
58.4
26.1
39.2
11.3
52.7
49.8
Charts and Diagram Understanding
General Visual Question Answering
General
Math
Reasoning
Multilingual
27.3
51.5
83.4
91.9
88.4
75.2
73.0
51.9
32.8
68.9
33.8
57.3
85.5
92.3
90.1
78.1
86.0
68.0
46.7
86.9
20.1
46.4
81.7
86.7
88.8
-
38.9
33.3
75.1
36.5
56.7
-
-
-
-
82.0
70.2
40.2
87.0