Prompts in Production

In production a prompt is code that runs millions of times on inputs nobody has seen, against a model that the provider will eventually change - so it needs templates, versioning, regression tests, monitoring and a defence against inputs that try to take it over.

Prompts as Code

flowchart LR
    A["✏️ Edit prompt<br/>(template in git)"] --> B["🧪 Offline eval<br/>golden set + CIs"]
    B -->|"no regression"| C["👀 Review"]
    C --> D["🚦 Canary / shadow<br/>5-10% traffic"]
    D --> E["🚀 Full rollout<br/>version pinned"]
    E --> F["📊 Monitor<br/>format rate, feedback,<br/>cost, drift"]
    F -.->|"failures become<br/>new test cases"| A
    B -->|"regression"| A

    style B fill:#d8dfe8,stroke:#b0bac8
    style D fill:#e8e0d4,stroke:#c8b89a
    style F fill:#dde4dc,stroke:#b0c4b0

Templates

Separate the fixed prompt from the variables, like a parameterised SQL query, and always delimit inserted content:

from jinja2 import Template

TICKET_PROMPT = Template("""Classify the support ticket below.
Categories:
{% for name, definition in categories.items() %}- {{ name }}: {{ definition }}
{% endfor %}
<ticket>
{{ ticket_text }}
</ticket>""")

prompt = TICKET_PROMPT.render(categories=CATEGORY_DEFS, ticket_text=ticket)

Versioning and registries

Store prompts in the repository next to the code that calls them (or in a registry that versions them), and record the prompt version, model version and parameters with every logged call so any output can be traced to exactly what produced it. Registries such as LangSmith, Langfuse, PromptLayer and MLflow's prompt registry add a UI, labels like production / staging, and links from each version to its eval results.

Model versions and drift

Providers update models behind aliases, and retire old snapshots on a published schedule. A prompt tuned on one version can behave differently on the next - different verbosity, stricter instruction following, a new tokenizer. Defences:

• Pin dated model snapshots in production; treat an alias like latest as a moving target.
• Treat a model change as a code change: run the eval set, re-tune the prompt if needed, canary it.
• Track provider deprecation dates so migrations are planned, not forced.
• Monitor format-compliance rate, output length, refusal rate, user feedback and cost per request - shifts without a code change point to the model.

Evaluation Gates

Every prompt or model change should pass an offline eval before it reaches users. The mechanics - building a golden set, choosing metrics, calibrating LLM judges and computing confidence intervals - are covered in the Evaluation module; the prompt-specific points are:

• Build the golden set from real traffic plus known hard cases and past failures; 100-500 items is typical for a single prompt.
• Compare paired: run old and new prompts on the same items and bootstrap the per-item difference. On small sets, apparent gains are often noise - the code lab shows a 4-point "improvement" whose interval spans zero.
• Gate on regressions per slice, not just the average - a prompt can improve overall while breaking one category.
• Use LLM-as-judge for open-ended outputs with a rubric and a judge validated against human labels; see LLM-as-Judge for its biases.

import random

def paired_gate(old_scores, new_scores, n=2000, max_drop=0.0):
    """Block the change if the 95% CI of (new - old) lies entirely below -max_drop."""
    diffs = [n_ - o for o, n_ in zip(old_scores, new_scores)]
    means = sorted(sum(random.choices(diffs, k=len(diffs))) / len(diffs) for _ in range(n))
    lo, hi = means[int(0.025 * n)], means[int(0.975 * n)]
    return hi >= -max_drop, (lo, hi)

Prompt Injection

Prompt injection is ranked first in the OWASP Top 10 for LLM Applications. The model reads instructions and data through the same channel - tokens - so text that looks like instructions can take over the model's behaviour.

Indirect injection is the more dangerous one, because the attacker never talks to your system - they only need to plant text where it will be read. It becomes critical when the model can act: an assistant that reads email and can send email can be told, by an email, to forward the inbox. The combination of private data, untrusted content and the ability to communicate externally is sometimes called the "lethal trifecta".

A layered defence

No wording in a prompt reliably stops injection - defences are probabilistic, and new attacks keep bypassing them. Design so that a successful injection can do little harm.

flowchart TD
    IN["📥 Untrusted input<br/>user text, documents, tool results"] --> L1
    L1["🧱 1. Separate and mark untrusted content<br/>tags, spotlighting (datamarking / encoding),<br/>'treat as data' instruction"] --> L2
    L2["🔍 2. Detect<br/>injection classifiers on inputs<br/>and on retrieved / tool content"] --> M
    M["🤖 Model trained on an instruction hierarchy"] --> L3
    L3["🔐 3. Limit privilege<br/>least-privilege tools, no secrets in context,<br/>human approval for consequential actions"] --> L4
    L4["🏗️ 4. Architecture<br/>keep untrusted content away from<br/>the component that plans actions"] --> L5
    L5["📊 5. Monitor output<br/>leaks, unexpected tool calls, URLs"] --> OUT(["✅ Response / action"])

    style L1 fill:#e8e2d9,stroke:#ccc4b8
    style L2 fill:#e8e0d4,stroke:#c8b89a
    style L3 fill:#d8dfe8,stroke:#b0bac8
    style L4 fill:#ddd8e4,stroke:#b8b0c8
    style L5 fill:#dde4dc,stroke:#b0c4b0

1. Mark untrusted content. Wrap it in tags and instruct the model to treat it as data. Spotlighting (Hines et al., 2024) strengthens this by transforming the untrusted text - interleaving a marker character between words ("datamarking") or base64-encoding it - which cut attack success rates from over 50% to under 2% in their experiments on some models, though not to zero.
2. Detect. Run a trained injection classifier on user input and on retrieved or tool content. Regex blocklists of phrases like "ignore previous instructions" catch only the laziest attacks and produce false positives - use them for logging, not as a defence.
3. Limit privilege. Give the model only the tools and data the task needs; keep secrets and credentials out of the context entirely; require human confirmation for irreversible or external actions (payments, emails, deletions).
4. Architect around it. Patterns such as keeping a "privileged" planner that never sees untrusted text, and a "quarantined" model that processes it without tool access, bound what an injection can reach. These are covered with agent security in Production Agents.
5. Monitor outputs. Flag responses that reproduce system-prompt text, contain unexpected URLs, or trigger unusual tool calls.

Assume the system prompt will leak eventually: never put secrets, keys or anything sensitive in it.

Safe Model or Prompt Migration

1. Run the golden set on the new model/prompt; compare paired, per slice.
2. Re-tune the prompt for the new model if needed (newer models often need less emphasis and fewer rules).
3. Re-run security tests: injection attempts, leak attempts, refusal behaviour.
4. Canary or shadow on a small share of traffic; watch format rate, feedback, cost and latency.
5. Roll out with the version pinned; keep the old version ready for rollback.

Check Yourself

Exercises

Study Notes

Must-know:

• Prompts are code: templates with delimited variables, version control or a registry, logged with model version and parameters
• Pin model snapshots; treat model upgrades as changes that go through eval and canary
• Gate changes with paired comparisons and CIs, per slice
• Prompt injection (OWASP LLM01): direct, indirect, leaking; indirect is the dangerous one for agents
• Layered defence: mark untrusted content (spotlighting), detect with classifiers, least privilege and human approval, architectural separation, output monitoring
• Never put secrets in prompts

References

• OWASP, Top 10 for LLM Applications (2025) - LLM01 Prompt Injection
• Greshake et al., Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (AISec 2023)
• Hines et al., Defending Against Indirect Prompt Injection Attacks With Spotlighting (2024)
• Wallace et al., The Instruction Hierarchy (2024)
• Willison, The lethal trifecta for AI agents (2025)
• Debenedetti et al., Defeating Prompt Injections by Design (CaMeL) (2025)

Last reviewed: 2026-09