Automated Prompt Optimization

Automated prompt optimization treats prompt text and few-shot examples as parameters to search over, scoring candidates with a metric on a training set - replacing manual trial and error with a reproducible optimization loop.

The Loop

Every method in this note is a variation on the same loop. What differs is how candidates are proposed.

flowchart LR
    P["🧪 Propose candidates<br/>instructions, examples"] --> E["📏 Evaluate<br/>metric on a training batch"]
    E --> S["🏆 Select / update<br/>keep the best, learn from scores<br/>or from failure traces"]
    S --> P
    S --> V["✅ Validate on held-out data<br/>before shipping"]

    style P fill:#e8e0d4,stroke:#c8b89a
    style E fill:#d8dfe8,stroke:#b0bac8
    style S fill:#dde4dc,stroke:#b0c4b0
    style V fill:#ddd8e4,stroke:#b8b0c8

GEPA's authors report that reflective prompt evolution can match or beat reinforcement-learning fine-tuning (GRPO) on several tasks with far fewer rollouts, and outperforms MIPROv2 - evidence that for many pipelines, optimizing the prompt is a cheaper first move than training weights.

DSPy: Programs, Not Prompts

DSPy (Khattab et al., 2024) separates what each LM call should do from how it is prompted. You declare signatures (typed inputs and outputs), compose modules into a program, and let an optimizer write the actual prompts and choose examples against your metric.

import dspy

dspy.configure(lm=dspy.LM("openai/<model>"))  # any LiteLLM-style "provider/model" string

class TicketClassifier(dspy.Signature):
    """Classify a customer-support ticket."""
    ticket: str = dspy.InputField()
    category: str = dspy.OutputField(desc="billing | bug | account_access | feature_request")

classify = dspy.ChainOfThought(TicketClassifier)   # adds a reasoning field before the output
print(classify(ticket="I was charged twice this month.").category)

A multi-step program is ordinary Python; retrieval is just a function you call:

class SupportAnswer(dspy.Module):
    def __init__(self, search):
        self.search = search                                         # any retriever function
        self.answer = dspy.ChainOfThought("context, question -> answer")

    def forward(self, question):
        context = self.search(question, k=5)
        return self.answer(context=context, question=question)

Optimizing it

trainset = [dspy.Example(ticket=t, category=c).with_inputs("ticket") for t, c in train_pairs]
valset   = [dspy.Example(ticket=t, category=c).with_inputs("ticket") for t, c in val_pairs]

def metric(gold, pred, trace=None):
    return gold.category == pred.category

optimizer = dspy.MIPROv2(metric=metric, auto="light")       # "light" | "medium" | "heavy" budget
optimized = optimizer.compile(classify, trainset=trainset, valset=valset)
optimized.save("ticket_classifier.json")                     # instructions + demos, versionable

# Reflective evolution: the metric can return feedback text as well as a score
def metric_with_feedback(gold, pred, trace=None, pred_name=None, pred_trace=None):
    ok = gold.category == pred.category
    feedback = "correct" if ok else f"expected {gold.category}, got {pred.category}"
    return dspy.Prediction(score=float(ok), feedback=feedback)

gepa = dspy.GEPA(metric=metric_with_feedback, auto="light",
                 reflection_lm=dspy.LM("openai/<strong-model>"))
evolved = gepa.compile(classify, trainset=trainset, valset=valset)

When DSPy fits

Pitfalls

• The metric is the specification. An optimizer maximises exactly what you measure. A metric that checks only format will produce well-formatted wrong answers; a lenient LLM judge will be gamed. Validate the metric on hand-labelled examples first.
• Overfitting. With a few dozen training examples, optimizers find prompts that fit those examples. Keep a held-out test set the optimizer never sees and report results on it - never on the training or validation set used during search.
• Model coupling. An optimized prompt is tuned to one model. Re-run optimization (not just evaluation) when you change models.
• Cost. Optimization runs hundreds or thousands of LM calls. Use the budget presets, cache LM calls, and use a cheaper model for trial runs.

Check Yourself

Exercises

Study Notes

Must-know:

• All methods: propose → evaluate on a metric → select; validate on held-out data
• APE (infer instructions), OPRO (score history), MIPROv2 (instruction + demo search with Bayesian optimization), GEPA (reflective evolution from traces and feedback), TextGrad (textual gradients)
• DSPy: signatures + modules + metric + optimizer; compiled programs can be saved and versioned
• The metric is the spec; overfitting and model coupling are the main risks
• Prompt optimization is often a cheaper first step than RL fine-tuning

References

• Zhou et al., Large Language Models Are Human-Level Prompt Engineers (APE) (ICLR 2023)
• Yang et al., Large Language Models as Optimizers (OPRO) (ICLR 2024)
• Khattab et al., DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (ICLR 2024)
• Opsahl-Ong et al., Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs (MIPRO) (EMNLP 2024)
• Agrawal et al., GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning (2025)
• Yuksekgonul et al., Optimizing generative AI by backpropagating language model feedback (TextGrad) (Nature, 2025)
• DSPy documentation

Last reviewed: 2026-09