Code Lab 02 - Terraform Private Endpoint and SLO Alerts

Build the two pieces of production plumbing that most LLM services need and most demos skip: private, least-privilege access from a workload to a managed model, written as infrastructure as code - and multi-window burn-rate alerts on the service's error budget. Both are tested offline: the infrastructure with tofu test against a mocked AWS provider, and the alerts with promtool test rules against synthetic traffic. No cloud account is needed to complete the lab.

← Back to Overview: Production Engineering · Concepts: Cloud Networking & IAM for AI · Infrastructure as Code · LLM Observability, SLOs & Incident Response

What's In This Lab

flowchart LR
    subgraph IAC["🏗️ terraform/"]
        TF["main.tf<br/>endpoint · SGs · role"] --> TT["🧪 tofu test<br/>mock AWS provider"]
    end
    subgraph SLO["🚨 alerts/"]
        R["slo-rules.yaml<br/>recording + burn-rate alerts"] --> PT["🧪 promtool test rules<br/>synthetic traffic"]
    end
    TT --> CI["✅ CI gate<br/>no cloud account needed"]
    PT --> CI

    style IAC fill:#d8dfe8,stroke:#b0bac8
    style SLO fill:#e8e0d4,stroke:#c8b89a
    style CI fill:#dde4dc,stroke:#b0c4b0

Run It

cd src/content/09-Production-Engineering/CodeLabs/02-Terraform-Private-Endpoint-and-SLO-Alerts

# Infrastructure (OpenTofu >= 1.8 or Terraform >= 1.7 for mock providers)
cd terraform
tofu init -backend=false
tofu fmt -check -recursive && tofu validate
tofu test

# Alerts (promtool ships with every Prometheus release)
cd ../alerts
promtool check rules slo-rules.yaml
promtool test rules slo-rules.test.yaml

Verified output:

tests/private_endpoint.tftest.hcl... pass
  run "endpoint_is_private_and_scoped"... pass
  run "role_requires_the_endpoint"... pass
  run "rejects_wildcard_models"... pass
  run "rejects_single_subnet"... pass

Success! 4 passed, 0 failed.

Checking slo-rules.yaml
  SUCCESS: 8 rules found

  SUCCESS                       # promtool test rules

To deploy for real, set vpc_id, private_subnet_ids, model_ids and cluster_name in a terraform.tfvars, configure a remote backend with locking (Infrastructure as Code), and run tofu plan with credentials. Load the rules into Prometheus (or a PrometheusRule resource for the Prometheus Operator) and have the gateway export llm_gateway_requests_total with a code label.

Walkthrough - What to Look At

1. Three layers of restriction on one call path. The security group decides who can reach the endpoint; the endpoint policy decides which actions and models can pass through it; the role policy decides what the workload may do and requires aws:SourceVpce, so the role's credentials are useless from anywhere else. Each layer limits the damage if another is misconfigured.
2. Policies as jsonencode, not data sources. Writing the policies inline keeps them plain HCL values the tests can jsondecode and inspect. It also matters for mocking: a mocked provider fills data-source outputs with random strings.
3. Validation is the cheapest test. model_ids rejects * and private_subnet_ids requires two subnets, so a bad change fails at plan - before review, and long before production. The expect_failures runs prove the validations actually trigger.
4. Mocks must still be valid. The first test run failed with Invalid ARN Value: the mock provider generated a random string for the role's ARN, and the Pod Identity resource validates ARN format. The mock_resource defaults supply realistic ARNs and ids - a reminder that mocks test your logic, not the cloud's behaviour.
5. Recording rules, then alerts. Each alert compares two precomputed error ratios - a long window that proves the burn is significant and a short one that stops the alert soon after recovery - against burn rate x (1 - SLO). With a 99.5% SLO those thresholds are 7.2%, 3% and 0.5%.
6. Test the silence too. Scenarios 3 and 4 assert that a healthy service doesn't page and that a slow 1% leak raises only a ticket. Alert rules that page on noise get ignored; tests that only check "it fires" can't catch that. A mutation check makes the point: raising the fast-burn threshold to 25x makes the 10% outage scenario fail.

Check Yourself

Exercises

References

• OpenTofu, Testing (tofu test, mock providers) (2026); HashiCorp, Terraform tests (2026)
• AWS, Use Amazon Bedrock with interface VPC endpoints (2026); EKS Pod Identity (2026)
• Prometheus, Unit testing for rules (2026)
• Beyer et al., The Site Reliability Workbook - Alerting on SLOs (Google, 2018)

Last reviewed: 2026-10