Getting Started with PDO (Prompt Duel Optimizer) with prompt-ops

llama-cookbook

Getting Started with PDO (Prompt Duel Optimizer) with prompt-opsDiscover how our next-generation prompt-optimization methods unlock new performance levels in Llama applications.Read more

Open on GitHub

Reading Time: 20 minutes

Difficulty: Intermediate to Advanced

What you'll learn:

How PDO uses competitive dueling to optimize prompts
The theory behind Thompson Sampling and Multi-Ranker Fusion
How to run PDO on your own datasets (hands-on with MS MARCO)
How to evaluate and interpret optimization results

This advanced tutorial will guide you through using PDO with prompt-ops to optimize prompts for Llama models through competitive dueling. We’ll cover:

1. Introduction to PDO
- The Problem PDO Solves
- Key Innovations
- Comparison with Other Methods
2. PDO Architecture Deep Dive
- Core Components
- Dueling Bandits Framework
- Thompson Sampling
- Multi-Ranker Fusion
3. Creating a PDO Project
- Project Structure
- Configuration Essentials
- Dataset Preparation
4. Running PDO Optimization
- The Optimization Process
- Understanding the Output
- Common Parameters
5. Analyzing Results
- Interpreting Optimized Prompts
- Performance Metrics
- Duel Statistics

Before we dive into the theory, a quick note: if you're the type who learns best by doing, we've prepared an interactive notebook that demonstrates PDO concepts through runnable code. Feel free to start there and reference back to this guide as needed.

1. Introduction to PDO

PDO (Prompt Duel Optimizer) is a novel prompt optimization technique introduced in the paper "Prompt Dueling Optimization: Tournament-Driven Prompt Discovery and Evaluation". Instead of scoring prompts individually, PDO treats optimization as a competitive tournament where prompts duel head-to-head to determine superiority.

Paper Reference: [arXiv:2510.13907](https://arxiv.org/abs/2510.13907) (2025)

Implementation: [meta-llama/prompt-ops](https://github.com/meta-llama/prompt-ops)

The Problem PDO Solves

Traditional prompt optimization methods face several critical challenges:

Absolute Scoring Bias: Single-point evaluation can be misleading - is a 0.85 score truly better than 0.82?
Limited Exploration: Greedy approaches miss superior prompts in unexplored regions
Ranking Uncertainty: Different metrics may rank prompts inconsistently
Exploitation vs Exploration: How to balance trying new prompts vs refining known good ones?

PDO addresses these limitations through a dueling bandit approach that treats prompt optimization as a competitive head-to-head tournament rather than absolute scoring.

Design Choice: Pairwise vs. Pointwise Evaluation

PDO uses pairwise comparison even when labels are available—trading computational efficiency for more robust prompt evaluation.

Key Research Innovations

The paper explores several novel concepts in prompt optimization

Dueling Bandits: Prompts compete in pairwise comparisons, not absolute rankings
Thompson Sampling: A smart exploration strategy that uses probability distributions to decide which prompt pairs to compare next. Learn more
Multi-Ranker Fusion: Combines multiple ranking algorithms (Copeland, Borda, Elo, TrueSkill) for robust evaluation
Variance-Driven Exploration: Selects opponent based on uncertainty, maximizing information gain
LLM-as-Judge: Uses language models to evaluate which prompt produces better responses
Adaptive Pruning: Removes consistently poor performers while maintaining diversity

PDO addresses these limitations through a dueling bandit approach that treats prompt optimization as a competitive head-to-head tournament rather than absolute scoring.

Design Choice: Pairwise vs. Pointwise Evaluation

PDO uses pairwise comparison even when labels are available—trading computational efficiency for more robust prompt evaluation.

Comparison with Other Methods

The paper compares PDO with several baseline approaches:

Method	Approach	Limitations	PDO's Advantage
Manual Prompting	Human-written instructions	Time-consuming, subjective	Automated competitive optimization
Few-shot Learning	Examples in prompt	Limited by context window	Optimizes instructions through duels
DSPy MIPRO	Bayesian-based optimization	Local optima, absolute scoring	Global exploration via bandits
OPRO	LLM-based optimization	Greedy, expensive	Efficient exploration via Thompson sampling

The Dueling Advantage

The paper’s key insight is that relative comparisons are more reliable than absolute scores. Dueling bandits are well-suited for prompt optimization because they:

Focus on what matters: “Which prompt is better?” not “What’s the exact score?”
Maintain diversity through probabilistic selection
Balance exploration (trying new prompts) and exploitation (refining winners)
Aggregate multiple ranking systems for robustness
Scale efficiently with the number of prompts

Visualizing the PDO Advantage

To understand why PDO’s dueling approach is superior, let’s compare it with traditional optimization:

The Traditional Problem:

When you score prompts independently (e.g., P4 gets 0.78, P5 gets 0.74), small differences might just be noise. Is P4 really better, or did it just get lucky on the test set? You can’t be sure.

PDO’s Solution:

Instead of asking “How good is this prompt?” (absolute), PDO asks “Which prompt is better?” (relative). By running head-to-head duels on the same examples, PDO eliminates scoring bias and reveals consistent winners.

The Visualization Below Shows:

Left: Traditional optimization relies on absolute scores that may be unreliable
Right: PDO builds a win matrix where each cell shows how often one prompt beats another in head-to-head duels

Key Insight: A prompt that wins 60%+ of duels against every opponent is a validated champion. This is PDO’s core innovation: reliable relative comparison through competition.

PDO discovers prompt quality through competitive duels,
✓ P4 dominates: wins 60%+ against ALL opponents (validated through competition)
where consistent winners emerge naturally from head-to-head comparisons.
\nKey Insight: P4’s superiority is validated across ALL matchups, not just by a single score that might be noise.

2. PDO Architecture Deep Dive

This section explores the core components of PDO as described in the research paper, focusing on the theoretical foundations and algorithmic innovations.

Core Components from the Paper

PDO consists of six main components:

Instruction Pool (𝒫): Collection of candidate prompts competing for superiority
Win Matrix (W): Tracks head-to-head results between all prompt pairs
Thompson Sampler: Selects duel pairs using probabilistic exploration
LLM Judge: Evaluates which prompt produces better responses
Multi-Ranker System: Aggregates multiple ranking algorithms for robust evaluation
Instruction Evolution: Generates new prompts by combining successful variants

Let’s explore each component in detail.

2.1. The Dueling Bandits Framework

The paper introduces PDO’s use of Dueling Bandits as the central optimization paradigm:

Traditional Multi-Armed Bandits:

Pull an arm → observe absolute reward
Problem: Reward scales may be unreliable or noisy

Dueling Bandits (PDO’s Approach):

Select two prompts → run head-to-head comparison
Observe relative preference: “Which is better?”
Advantage: Relative comparisons are more stable than absolute scores

none
Initialize:
          - Start with base prompt + generate initial variations
          - Initialize Win matrix W (all zeros)
          For each round t = 1 to T:
          1. Use Thompson Sampling to select prompt pair (i, j)
          2. Run duel: Both prompts answer the same examples
          3. LLM judge decides winner on each example
          4. Update W[i,j] with wins, W[j,i] with losses
          5. Compute multi-ranker scores (Copeland, Borda, Elo, TrueSkill)
          6. [Optional] Generate new prompts by combining top performers
          7. [Optional] Prune worst-performing prompts
          Return:
          - Best prompt according to aggregated rankings

Key Innovation: The Win matrix W becomes increasingly informative, guiding both exploration (via Thompson sampling) and exploitation (via ranking systems).

2.2. Thompson Sampling

One of PDO’s most sophisticated components is its Double Thompson Sampling strategy:

What is Thompson Sampling?

Thompson Sampling is a probabilistic approach to the exploration-exploitation dilemma:

Maintain a belief distribution about each prompt’s quality
Sample from these distributions to select actions
Naturally balances trying new things vs. exploiting known winners

PDO’s Double Thompson Sampling

The paper describes a two-stage selection process:

Stage 1: Select First Prompt
- For each prompt pair (i,j), model win probability as Beta(wins+1, losses+1)
- Sample a win-rate matrix θ from these Beta distributions
- Compute multiple rankings from θ (Copeland, Borda, win-rate)
- Combine with Elo and TrueSkill ratings using Dirichlet-weighted fusion
- Apply softmax with temperature τ for final selection
Stage 2: Select Second Prompt (Opponent)
- Among remaining prompts, filter to those still plausibly competitive
- Select the one with maximum variance in win probability vs. first prompt
- Rationale: High variance = high uncertainty = maximum information gain from this duel

Benefits:

Efficient exploration: Doesn’t waste duels on clearly inferior prompts
Information-driven: Prioritizes duels that reduce uncertainty
Adaptive: Automatically shifts from exploration to exploitation as data accumulates

Understanding the Thompson Sampling Visualization

The diagram below shows Beta distributions representing our belief about each prompt’s win probability. This is the heart of how PDO decides which prompts to compare in duels:

What You’re Looking At:

Each plot shows a probability distribution over win rates (0 to 1 on x-axis)
The height (density) represents how confident we are about that win rate
The width represents uncertainty – wider = less certain

The Three Scenarios:

Prompt A (Strong Performer - Green): 15 wins, 5 losses
Narrow, tall distribution centered at 0.73
High confidence: We’re pretty sure this prompt wins ~73% of the time
PDO interpretation: Reliable choice, but already well-understood
Prompt B (Average - Blue): 10 wins, 10 losses
Medium width distribution centered at 0.50
Moderate confidence: Wins about half the time
PDO interpretation: Middle-of-the-pack, not particularly interesting
Prompt C (Uncertain - Orange): 2 wins, 2 losses
Wide, flat distribution centered at 0.50
HIGH UNCERTAINTY: Could be anywhere from 0.2 to 0.8!
PDO interpretation: High information value – might be secretly excellent OR terrible

PDO’s Smart Strategy:

First prompt selection: Sample from all distributions → A likely selected (highest mean)
Second prompt selection: Pick the opponent with maximum variance vs. the first
- Dueling A vs B tells us little (both well-understood)
- Dueling A vs C tells us a LOT (huge uncertainty about C)

The Key Insight: By choosing C as the opponent, PDO maximizes information gain. After this duel:

If C wins → Great! We discovered a hidden champion
If C loses → Good! We eliminated uncertainty and can focus elsewhere

This is why PDO converges faster than naive approaches – it strategically explores high-uncertainty regions rather than randomly sampling prompts.

2.3. Multi-Ranker Fusion

One of PDO’s most important innovations is using multiple ranking algorithms simultaneously to create a more robust evaluation system.

What is Multi-Ranker Fusion?

Imagine you’re trying to determine the best chess player. Would you:

Option A: Use only one ranking system (e.g., just count wins)
Option B: Combine multiple perspectives (wins, strength of opponents beaten, consistency, etc.)

PDO chooses Option B – it fuses (combines) the opinions of 5 different ranking algorithms, each with its own strengths and blind spots. This is called Multi-Ranker Fusion.

The Core Problem: No single ranking system is perfect. Each has biases:

Some favor consistent performers
Some reward crushing weak opponents
Some value beating strong opponents highly
Some are sensitive to the order of matches

By combining all 5, PDO gets a consensus view that’s more reliable than any individual ranker.

Why Multiple Rankers?

Different ranking systems capture different aspects of prompt quality:

Ranking System	What It Measures	Strengths	Weaknesses
Copeland	Number of opponents beaten	Simple, intuitive	Ignores margin of victory
Borda	Sum of win probabilities	Accounts for all matchups	Can be dominated by many weak wins
Average Win Rate	Mean win probability	Easy to interpret	Doesn't account for opponent strength
Elo	Chess-style rating	Accounts for opponent strength	Sensitive to order of matches
TrueSkill	Bayesian skill rating	Confidence intervals, robust	More complex, computational cost

Fusion Strategy

The paper describes a sophisticated fusion approach:

none
For each round:
          1. Compute all 5 ranking scores for each prompt
          2. Normalize each to [0, 1] range
          3. Sample fusion weights from Dirichlet(1,1,1,1,1)
          → Introduces randomness in how rankings are combined
          4. Compute weighted combination: score = Σ(weight_i × ranking_i)
          5. Use fused scores for prompt selection

What’s Happening Here?

Step 3 is key: Instead of fixed weights (e.g., always 20% each), PDO randomly samples weights each round.
Dirichlet distribution ensures weights sum to 1.0 but vary each time.
Example: One round might weight Elo heavily (0.4, 0.15, 0.15, 0.15, 0.15), next round Copeland (0.15, 0.4, 0.15, 0.15, 0.15).
This explores different “philosophies” of ranking over time.

Benefits:

Robustness: No single ranking system dominates
Exploration: Dirichlet sampling explores different ranking perspectives
Consensus: Prompts that rank well across ALL systems are truly superior

Concrete Example

Let’s say we have 3 prompts after some duels:

Prompt	Copeland	Borda	Win Rate	Elo	TrueSkill	Average
P1	1st	1st	1st	2nd	1st	1st (winner!)
P2	2nd	2nd	3rd	1st	2nd	2nd
P3	3rd	3rd	2nd	3rd	3rd	3rd

Prompt

Copeland

Borda

Win Rate

Elo

TrueSkill

Average

1st

2nd

1st

1st (winner!)

2nd

3rd

1st

2nd

2nd

3rd

2nd

3rd

3rd

Key Insight: P1 wins 4 out of 5 ranking systems – it’s a consensus champion. Even though P2 tops the Elo ranking (maybe it beat one strong opponent), P1 is more consistently excellent across all evaluation criteria.

This is why Multi-Ranker Fusion is powerful: It prevents a prompt from gaming one specific metric and ensures true, well-rounded superiority.

4. LLM-as-Judge

The paper emphasizes that PDO’s evaluation mechanism is crucial for effective optimization:

Traditional Metrics: Hard-coded rules (exact match, F1, BLEU, etc.)

PDO’s LLM Judge:

Natural Language Evaluation: Can assess nuanced qualities (helpfulness, clarity, correctness)
Pairwise Comparison: Given two responses, which is better and why?
Flexible Criteria: Can evaluate domain-specific quality without custom metric code
Reasoning Output: Provides explanation for decisions (useful for debugging)

Judge Prompt Structure

python
You are an expert evaluator. Compare these two responses:
          Input: {question}
          Expected Answer: {label}
          Response A:
          {response_from_prompt_A}
          Response B:
          {response_from_prompt_B}
          Which response is better? Consider:
          - Correctness: Does it match the expected answer?
          - Completeness: Does it address all aspects?
          - Clarity: Is it well-structured and understandable?
          Output format:
          {
          "reasoning": "Your detailed comparison",
          "winner": "A" or "B"
          }

Benefits of LLM-as-Judge:

Works for any task (classification, generation, reasoning)
Captures subtle quality differences
Scales to new domains without custom metric engineering

3. Creating a prompt-ops PDO Project

Now that we understand the theoretical foundations, let’s create a real PDO project using the MS MARCO dataset as our example, you can also follow along this hands-on section here: Github notebook.

Why MS MARCO?

MS MARCO (Microsoft MAchine Reading COmprehension) is an excellent dataset for demonstrating PDO because:

It’s a real-world open-ended QA task
Requires nuanced understanding and clear communication
Benefits from prompt optimization for answer quality
Shows measurable improvements with PDO

Getting Started with the MS MARCO Example

We’ll use a pre-configured MS MARCO PDO project that demonstrates best practices:

Installing prompt-ops

Before we can work with the MS MARCO example, we need to install prompt-ops. You can install it from PyPI or from source for the latest features.

Install from Source

python
# Install from source
            # Clone and install in development mode (editable install)
            !git clone -b pdo-notebook https://github.com/meta-llama/prompt-ops.git
            !cd prompt-ops && pip install -e .

Now let’s get the MS MARCO Example

With prompt-ops installed, we can now clone and explore the MS MARCO PDO example:

python
# Clone the pre-configured MS MARCO PDO example
            !cd prompt-ops/use-cases/ms-marco-pdo && ls -la

Understanding the MS MARCO Configuration

Let’s examine the key configuration elements for this open-ended QA task:

MS MARCO PDO Configuration

python
# PDO Configuration for MS MARCO Open-Ended QA
            name: ms-marco-qa
            models:
            task_model: "openrouter/meta-llama/llama-3.3-70b-instruct"      # Model to optimize
            proposer_model: "openrouter/meta-llama/llama-3.3-70b-instruct" # For evolution & judging
            provider: "openrouter"
            temperature: 0.0                                                # Zero temp for consistency
            dataset:
            path: "dataset/ms_marco_description.json"
            input_field: "question"                                        # Direct field (not nested)
            golden_output_field: "answer"                                  # Ground truth answers
            optimization:
            strategy: "pdo"                                               # PDO strategy
            task_type: "open_ended"                                       # Open-ended QA task
            # Judge rubric aligned with final evaluation criteria
            judge_requirement: |
            - Accuracy: Correct and contextually faithful.
            - Relevance: Stays on-topic.
            - Clarity: Clear and well-structured.
            - Conciseness: Uses only necessary information; avoids unnecessary detail.
            # Core dueling parameters
            total_rounds: 30                                              # Number of optimization rounds
            num_duels_per_round: 3                                        # Duels per round
            num_eval_examples_per_duel: 10                                # Examples per duel
            num_initial_instructions: 2                                   # Start with 2 prompts
            # Instruction evolution
            num_top_prompts_to_combine: 2                                 # Top K for combination
            num_new_prompts_to_generate: 1                                # New prompts per gen round
            max_prompts_to_generate: 10                                   # Max total prompt pool size
            gen_new_prompt_round_frequency: 2                             # Generate every 2 rounds
            # Execution parameters
            max_concurrent_threads: 10                                    # Parallel API calls
            verbose: true                                                 # Detailed logging
            prompts:
            system: "prompts/prompt.txt"                                  # Initial prompt file
            metric:
            class: "llama_prompt_ops.core.metrics.LLMJudgeStandardMetric"

Key Differences for Open-Ended Tasks

task_type: ‘open_ended’ - Enables LLM-based evaluation
judge_requirement - Custom rubric for answer quality
No strict_json - Answers are free-form text
Smaller num_eval_examples_per_duel - Quality over quantity

Support both close-ended and open-ended task

For close-ended (classification) tasks, set optimization.task_type to close_ended in your configuration, and specify optimization.answer_choices (such as ["Yes", "No"]), though this defaults to yes/no if omitted. In this mode, the model returns its answer as a JSON object that includes both reasoning and the chosen answer, and judging uses a fixed rubric—evaluating both correctness and reasoning quality.

python
optimization:
            strategy: "pdo"
            task_type: close_ended
            answer_choices: ["Yes", "No"]

For open-ended tasks (meaning any task that isn’t simple classification), set optimization.task_type to open_ended in your configuration. You can also provide a custom evaluation rubric by specifying optimization.judge_requirement. If you omit this, PDO will automatically generate appropriate judging criteria based on your dataset summary and sample questions. In open-ended mode, the model generates free-form answers, and the judge selects the better response using either your custom rubric or the automatically generated one, providing both an explanation and a winner. For example, if you are working on a summarization task, your YAML might look like this with a specific customized requirement:

python
optimization:
            strategy: "pdo"
            task_type: open_ended
            judge_requirement: |
            - Key points coverage (40%): Captures major decisions, actions, owners, dates.
            - Faithfulness (30%): No invented details or attributions.
            - Structure (15%): Logical, grouped bullets or sections.
            - Brevity & clarity (10%): Minimal fluff; readable.
            - Tie-Breaker (5%): Better prioritization of actionable items.

Dataset Preparation

The MS MARCO dataset included in our example contains curated question-answer pairs. PDO works effectively with:

Moderate dataset sizes (50-200 examples): Our subset has 100 QA pairs
Clear input/output pairs: Questions with human-written answers
Diverse examples: Covers various topics and question types

Let’s look at the MS MARCO data structure:

MS MARCO Dataset Structure

json
{
            "question": "what is the purpose of an mri",
            "answer": "Magnetic resonance imaging (MRI) is a medical imaging technique used to visualize internal structures of the body in detail. MRI makes use of the property of nuclear magnetic resonance (NMR) to image nuclei of atoms inside the body. An MRI scan can be used as an extremely accurate method of disease detection throughout the body and is most often used after other testing fails to provide sufficient information to confirm a patient's diagnosis."
            }

Dataset Characteristics

Direct question-answer pairs
Questions are natural language queries
Answers are comprehensive, factual explanations
No additional context needed (unlike reading comprehension)

This structure is perfect for PDO because:

Clear evaluation criteria (answer quality)
Natural variation in question types
Human-written gold standard answers

Initial Prompt Template

For MS MARCO, we start with a simple, neutral prompt that PDO will optimize:

Design principles:

Keep it simple - Let PDO discover complexity
Domain-neutral - Avoid biasing toward specific answer styles
Clear task definition - The model knows it should answer questions

PDO will evolve this through competitive testing to discover:

Optimal answer length and detail level
Best communication style
Most effective structure

MS MARCO Initial Prompt

prompts/prompt.txt:

python
initial_prompt = """You are an expert answerer. 
            Read the question and the provided context (if any).
            Write a concise, accurate answer in your own words.
            """

python
initial_prompt = """You are an expert answerer. 
            Read the question and the provided context (if any).
            Write a concise, accurate answer in your own words.
            """

Why Start Simple?

Establishes baseline performance
No assumptions about optimal style
PDO will discover what works through duels

What PDO Might Discover:

Whether step-by-step explanations help
Optimal level of technical detail
Best way to structure complex answers
Tone and communication style

Example Evolution:

Round 1: Basic prompt (above)

Round 5: Adds ‘provide clear explanations’

Round 10: Discovers ‘step-by-step’ improves clarity

Round 20: Adds role specification (‘knowledgeable researcher’)

Round 30: Converges on optimal prompt through competition

Judge requirement and final metric alignment

For the evaluation of open-ended tasks, an increasing number of benchmarks now rely on LLM judges due to their flexibility and the availability of more capable frontier or fine-tuned models. In MS MARCO, we also use an LLM judge to evaluate the outputs generated by the model.

The final evaluation of a prompt involves comparing the LLM-generated response, produced in response to the instruction prompt, with the ground-truth answer using an LLM judge template as shown below.

python
JUDGE_PROMPT = """Begin your evaluation by carefully comparing the AI-generated answer with the reference solution. Identify any outputs that aren't true, as well as omissions, or deviations, and simulate human judgments in explaining how these impact the overall quality of the response. Ensure that your assessment is objective and consistent.
            At the end of your evaluation, assign a score from 1 to 5 based on the following scale:
            - 1: Very poor—does not meet the requirements or is significantly incorrect.
            - 2: Poor—contains major errors or omissions.
            - 3: Fair—adequate but with notable flaws.
            - 4: Good—meets the requirements with minor errors.
            - 5: Excellent—fully accurate and well-articulated.
            [User Question]
            {question}
            [Reference Solution]
            {ground_truth}
            [AI-Generated Answer]
            {prediction}
            Your response should be a valid json string (no backticks) following this schema:
            {{
            "explanation": "{{Detailed reasoning based on the comparison}}",
            "score": {{1-5}}
            }}
            """

4. Running PDO Optimization

Now let’s run PDO on the MS MARCO dataset and watch the optimization process in action.

Prerequisites

Before running PDO, ensure you have:

API Key: Set your OpenRouter API key
Project Setup: Cloned the MS MARCO example
Dependencies: Installed prompt-ops

shell
# Set your API key (or add to .env file)
            export OPENROUTER_API_KEY="your-key-here"
            # Or if using a .env file
            echo "OPENROUTER_API_KEY=your-key-here" > .env

Understanding the Judge

For open-ended tasks like MS MARCO, PDO uses an LLM judge to compare answers. The judge evaluates based on the criteria in judge_requirement:

Accuracy: Is the answer factually correct?
Relevance: Does it address the question?
Clarity: Is it well-structured and understandable?
Conciseness: Does it avoid unnecessary detail?

This aligns with how we’ll evaluate the final optimized prompt

Running PDO

python
!cd prompt-ops/use-cases/ms-marco-pdo && prompt-ops migrate --config config.yaml

After running PDO, we store the optimized prompt as follows, ready for final evaluation.

python
OPTIMIZED_PROMPT = (
            """Embody the role of a knowledgeable researcher and provide a clear, 
            step-by-step explanation to answer the user's question, utilizing the provided 
            context and avoiding technical jargon unless absolutely necessary, to ensure a 
            concise and accurate response."""
            )

5. Evaluate the optimized prompt with LLM judge

In this section, we evaluate the performance of the optimized prompt after running PDO against the initial prompt. We compare generated answers against ground truth using the judge prompt from Section 1 (returns JSON with explanation + 1–5 score).

shell
def judge_scores(preds: list[str]):
            """Judge with guided JSON decoding (fallback to free-form if unsupported)."""
            import json as _json
            import numpy as _np
            scores = []
            explanations = []
            judge_prompts = []
            for pred, r in zip(preds, rows):
            judge_prompts.append(
            JUDGE_PROMPT.format(
            question=r["question"],
            ground_truth=r["answer"],
            prediction=pred,
            )
            )
            response_format = {
            "type": "json_schema",
            "json_schema": {
            "name": "judge_score",
            "schema": {
            "type": "object",
            "properties": {
            "explanation": {
            "type": "string",
            "description": "Detailed reasoning based on the comparison",
            },
            "score": {
            "type": "integer",
            "description": "Score from 1 to 5",
            "enum": [1, 2, 3, 4, 5],
            },
            },
            "required": ["explanation", "score"],
            "additionalProperties": False,
            },
            "strict": True,
            },
            }
            try:
            outs = model.generate_batch(
            judge_prompts, max_threads=10, temperature=0.0, response_format=response_format
            )
            except Exception:
            outs = model.generate_batch(
            judge_prompts, max_threads=10, temperature=0.0
            )
            for o in outs:
            obj = _try_json_parse(o)
            score = obj.get("score", 0)
            exp = obj.get("explanation", "")
            try:
            score = float(score)
            except Exception:
            score = 0.0
            scores.append(score)
            explanations.append(exp)
            return _np.array(scores), explanations

shell
import os, json
            import numpy as np
            from pathlib import Path
            import sys
            sys.path.append(os.path.abspath("../../src"))
            from prompt_ops.core.model import setup_model
            # Load dataset
            DATA_PATH = "prompt-ops/use-cases/ms-marco-pdo/dataset/ms_marco_description.json"
            rows = json.load(open(DATA_PATH, "r"))
            # Model for answering and for judging (same adapter; different prompts)
            model = setup_model(
            adapter_type="litellm",
            model_name="openrouter/meta-llama/llama-3.3-70b-instruct",
            api_base=os.environ.get("OPENROUTER_API_BASE", "https://openrouter.ai/api/v1"),
            api_key=os.environ.get("OPENROUTER_API_KEY"),
            max_tokens=4096,
            temperature=0.0,
            )
            # Helper
            def _try_json_parse(s: str):
            try:
            start = s.find("{"); end = s.rfind("}") + 1
            blob = s[start:end] if (start >= 0 and end > start) else s
            return json.loads(blob)
            except Exception:
            return {}
            # Baselines and PDO instruction
            BASELINES = {
            "Initial Prompt": """You are an expert answerer. 
            Read the question and the provided context (if any).
            Write a concise, accurate answer in your own words.
            """,
            }
            def predict_answers(instruction: str):
            prompts = []
            for r in rows:
            q = r["question"].strip()
            prompts.append(f"{instruction}\n\nQuestion:\n{q}")
            return model.generate_batch(prompts, max_threads=10, temperature=0.0)
            results = {}
            # Evaluate baselines
            for name, instr in BASELINES.items():
            print(f"evaluating... {name}")
            preds = predict_answers(instr)
            s, _ = judge_scores(preds)
            results[name] = float(np.mean(s)) if len(s) else 0.0
            # Evaluate PDO (if you paste the optimized instruction)
            if OPTIMIZED_PROMPT:
            print("evaluating... PDO")
            preds = predict_answers(OPTIMIZED_PROMPT)
            s, _ = judge_scores(preds)
            results["PDO"] = float(np.mean(s)) if len(s) else 0.0
            print("\n=== LLM Judge Avg Score (1–5) on MS MARCO Description ===")
            for k, v in results.items():
            print(f"{k}: {v:.2f}")

PDO achieves a higher average LLM judge score (4.64) than the Initial Prompt (4.48) on the MS MARCO Description, indicating a measurable improvement in answer quality.

python
import matplotlib.pyplot as plt
            plt.figure(figsize=(8, 4.5))
            bars = plt.bar(results.keys(), results.values(), color="#4e79a7", edgecolor="#2e4a62", linewidth=1.5, width=0.55)
            plt.title("LLM Judge Avg Score (1–5) on MS MARCO Description", fontsize=15, fontweight="bold", pad=14)
            plt.ylabel("Avg Score (1–5)", fontsize=13)
            plt.ylim(4, 5.1)
            plt.xticks(rotation=15, fontsize=11)
            plt.yticks(fontsize=11)
            plt.grid(axis='y', linestyle='--', alpha=0.25)
            for bar, v in zip(bars, results.values()):
            plt.text(bar.get_x() + bar.get_width() / 2, v + 0.10, f"{v:.2f}", ha="center", va="bottom", fontsize=12, fontweight="bold", color="#34495e")
            ax = plt.gca()
            ax.spines['top'].set_visible(False)
            ax.spines['right'].set_visible(False)
            plt.tight_layout()
            plt.show()

Conclusion and Key Takeaways

Summary of PDO’s Innovations

Based on our exploration of the PDO paper and practical implementation:

Dueling Bandits Framework: PDO treats optimization as a tournament where prompts compete head-to-head, making relative comparisons more reliable than absolute scoring.
Thompson Sampling: Smart exploration strategy that balances trying new prompts (exploration) with refining known winners (exploitation) through probabilistic sampling.
Multi-Ranker Fusion: Combines 5 different ranking algorithms (Copeland, Borda, Win Rate, Elo, TrueSkill) with Dirichlet-weighted fusion for robust evaluation.
LLM-as-Judge: Uses language models to evaluate pairwise preferences, enabling nuanced quality assessment beyond hard metrics.
Instruction Evolution: Generates new prompts by combining top performers, allowing discovery of superior variants through competitive pressure.
Adaptive Pruning: Removes consistently poor performers while maintaining diversity in the prompt pool.

When to Use PDO

The paper suggests PDO is particularly effective for:

High-stakes applications where prompt quality is critical
Complex tasks with nuanced quality criteria
Scenarios where absolute scoring is unreliable (subjective tasks)
Budget available for thorough exploration (more LLM calls than simpler methods)

Paper’s Key Results

The paper demonstrates PDO’s effectiveness:

Outperforms gradient-based methods: More thorough exploration avoids local optima
Robust rankings: Multi-ranker consensus prevents over-reliance on single metric
Efficient evolution: Thompson sampling focuses computational budget on informative duels
Interpretable results: Win matrices and Elo ratings provide clear understanding of prompt quality

Final Thoughts

PDO represents an important advance in prompt optimization by bringing dueling bandit algorithms to LLM prompt engineering. Its competitive approach, combined with sophisticated exploration strategies and multi-ranker fusion, creates a powerful framework for discovering high-quality prompts through natural selection in a tournament setting.

The key insight is that asking “which is better?” is often more reliable than asking “how good is this?” – PDO leverages this principle throughout its design.

For more details, refer to the full paper and implementation documentation.

Additional Resources

References

PDO Paper: “Prompt Dueling Optimization: Tournament-Driven Prompt Discovery and Evaluation” (arXiv:2510.13907)

Tips for Success

Start small: Use 10-30 training examples as recommended by the paper
Design good feedback: Specific, actionable feedback functions are crucial
Monitor validation: Track validation performance to ensure generalization
Experiment with budgets: Start with 500-1000 metric calls
Enable merging: Set use_merge: true for best results

This tutorial covered the theoretical foundations and practical implementation of PDO based on dueling bandits and the research paper. For hands-on practice, try the web of lies tutorial or adapt PDO to your own classification or generation tasks.

Stay up-to-date

Our latest updates delivered to your inbox

Subscribe to our newsletter to keep up with the latest AI updates, releases and more.

Table of Contents