Spatial IaC Terraform & Pulumi for GIS

SLO-Driven Alerting for WMS and WMTS Endpoints

An OGC endpoint has an unusually deceptive availability profile. GetCapabilities answers in milliseconds from a cached document whether or not a single layer can actually be rendered, so a WMS whose data store connection has failed will report 100 per cent availability to any monitor that probes the obvious endpoint. Meanwhile GetMap at a dense extent may take four seconds legitimately, and a threshold set for the capabilities response will page continuously. Alerting on these services means measuring the operations users actually depend on, at request shapes that reflect real maps. This guide extends Observability for Spatial Infrastructure within CI/CD Automation and Governance.

Choosing the indicators

Three service level indicators cover an OGC endpoint properly, and the first one is the one usually missing.

Rendered-response success. The fraction of GetMap and GetTile requests that return an image of the expected content type with a non-trivial body. The qualifier matters: WMS reports errors as images when EXCEPTIONS=INIMAGE is set, so a 200 response carrying a picture of an error message satisfies any status-code check and is a total failure from the user’s point of view. Measuring the response size distribution catches it, because an error image is small and uniform.

Render latency at a realistic request shape. Measured on GetMap at extents and layer combinations that reflect actual client behaviour, split by layer class. A cached basemap and a live rendered operational layer need separate objectives, because their honest budgets differ by an order of magnitude and a single number will be wrong for both.

Capabilities freshness. GetCapabilities should list the layers that are actually publishable. A capabilities document advertising a layer that fails to render is worse than one that omits it, because a client will bind to it.

Two ways a broken WMS reports as healthy A monitor probing GetCapabilities receives a fast successful response served from a cached document, which says nothing about whether any layer can actually be rendered — a WMS whose data store connection has failed answers this endpoint perfectly. A monitor probing GetMap with the exceptions parameter set to render errors inside the image receives a 200 response carrying a valid image that happens to be a picture of an error message. Both probes report full availability while the service renders nothing usable. Detecting the second requires checking the response body size distribution, because an error image is small and uniform where a real map tile is neither. Probe GetCapabilities fast 200, cached document Reports healthy with every data store down Probe GetMap, status only EXCEPTIONS=INIMAGE Reports healthy serving a picture of an error What actually works probe GetMap check content type check body size an error image is small An error image is small and uniform; a real map tile is neither. The size distribution is the detector. Set EXCEPTIONS=XML on synthetic probes so a failure is a failure rather than a picture of one.

Prerequisites and environment assumptions

Terraform 1.6 or later with hashicorp/aws pinned at ~> 5.60. Access logs or request metrics from the load balancer or CDN with response size available. A synthetic probe capability, because passive metrics alone cannot tell you a layer renders correctly when nobody is requesting it — a layer used weekly needs a probe or its failure is discovered by the person who needed it.

Agree the objectives with the people who depend on the service before writing any alarm. An objective invented by the platform team is a number the platform team defends; one agreed with the cartography and application teams is a shared commitment, and the difference shows the first time an error budget runs low and someone has to decide whether to pause a release.

Step-by-step implementation

  1. Define the indicators per layer class. Group layers into two or three classes with genuinely different budgets — cached basemap, live operational, heavy analytical — and set an objective per class. Per-layer objectives are unmaintainable; a single global objective is dishonest.

  2. Instrument the success indicator on content, not status. Emit a metric from log analysis that counts a request as successful only when the content type is an image and the body size exceeds an error-image threshold. This is the single most valuable instrumentation change on an OGC endpoint.

  3. Add synthetic probes for low-traffic layers with EXCEPTIONS=XML. Forcing exceptions into XML rather than into the image means a probe failure is an actual failure rather than a successfully delivered picture of one.

  4. Compute the error budget and alert on burn rate, not on instantaneous failure. A short spike that consumes a per cent of the monthly budget is noise; a sustained rate that would exhaust it in a day is an incident. Two burn-rate alarms — a fast one for severe burns and a slow one for grinding degradation — cover both without paging on every blip.

  5. Gate deploys on remaining budget. When the budget falls below a threshold, only reliability changes ship. Expressed in the pipeline described in Pipeline Orchestration for Spatial Deploys, this is what turns an objective from a report into a control.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.60" }
  }
}

variable "oncall_topic_arn" { type = string }
variable "runbook_url" { type = string }

# Success is CONTENT-based: an image content type and a body larger than an
# error image. A status-code metric would count a rendered error as a success.
resource "aws_cloudwatch_log_metric_filter" "getmap_success" {
  name           = "wms-getmap-success"
  log_group_name = aws_cloudwatch_log_group.wms_access.name
  pattern = "{ ($.request = \"*GetMap*\") && ($.status = 200) && ($.bytes > 4096) && ($.content_type = \"image/*\") }"

  metric_transformation {
    name      = "GetMapSuccess"
    namespace = "SpatialPlatform/OGC"
    value     = "1"
    dimensions = {
      LayerClass = "$.layer_class"
    }
  }
}

resource "aws_cloudwatch_log_metric_filter" "getmap_total" {
  name           = "wms-getmap-total"
  log_group_name = aws_cloudwatch_log_group.wms_access.name
  pattern        = "{ $.request = \"*GetMap*\" }"

  metric_transformation {
    name      = "GetMapTotal"
    namespace = "SpatialPlatform/OGC"
    value     = "1"
    dimensions = {
      LayerClass = "$.layer_class"
    }
  }
}

locals {
  slo_target       = 0.995      # 99.5% of GetMap requests render an actual map
  budget_fraction  = 1 - local.slo_target
  # Fast burn: consuming 2% of a 30-day budget in an hour would exhaust it in
  # about a day. That is an incident. Slow burn catches grinding degradation.
  fast_burn_factor = 14.4
  slow_burn_factor = 6.0
}

resource "aws_cloudwatch_metric_alarm" "fast_burn" {
  alarm_name          = "wms-slo-fast-burn"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  threshold           = local.budget_fraction * local.fast_burn_factor

  metric_query {
    id          = "burn"
    expression  = "1 - (success / IF(total > 0, total, 1))"
    label       = "error rate over 1h"
    return_data = true
  }
  metric_query {
    id = "success"
    metric {
      namespace   = "SpatialPlatform/OGC"
      metric_name = "GetMapSuccess"
      period      = 3600
      stat        = "Sum"
      dimensions  = { LayerClass = "live-operational" }
    }
  }
  metric_query {
    id = "total"
    metric {
      namespace   = "SpatialPlatform/OGC"
      metric_name = "GetMapTotal"
      period      = 3600
      stat        = "Sum"
      dimensions  = { LayerClass = "live-operational" }
    }
  }

  alarm_description = "Burning the GetMap error budget fast enough to exhaust it in ~1 day. Runbook: ${var.runbook_url}"
  alarm_actions     = [var.oncall_topic_arn]
  treat_missing_data = "notBreaching"
  tags = { Owner = "geospatial-platform", Runbook = var.runbook_url }
}

# Low-traffic layers have no passive signal at all: without a probe, a broken
# layer is discovered by whoever needed it.
resource "aws_synthetics_canary" "getmap_probe" {
  name                 = "wms-getmap-probe"
  artifact_s3_location = "s3://${aws_s3_bucket.canary.id}/wms/"
  execution_role_arn   = aws_iam_role.canary.arn
  runtime_version      = "syn-nodejs-puppeteer-9.0"
  handler              = "probe.handler"
  zip_file             = data.archive_file.probe.output_path

  schedule { expression = "rate(5 minutes)" }
}
// probe.js — EXCEPTIONS=XML so a failure is a failure, not a picture of one.
const synthetics = require('Synthetics');

const LAYERS = ['parcels', 'zoning', 'flood_extent'];
const BBOX = '-122.5,37.7,-122.3,37.9';

exports.handler = async function () {
  for (const layer of LAYERS) {
    const url = `https://maps.example.org/geoserver/wms?service=WMS&version=1.3.0`
      + `&request=GetMap&layers=${layer}&bbox=${BBOX}&width=512&height=512`
      + `&crs=EPSG:4326&format=image/png&EXCEPTIONS=XML`;

    const res = await synthetics.executeHttpStep(`getmap-${layer}`, url);

    // Content type AND size. A rendered error image would pass a status check
    // and, on some servers, even a content-type check.
    if (!res.headers['content-type'].startsWith('image/')) {
      throw new Error(`${layer}: not an image (${res.headers['content-type']})`);
    }
    if (Number(res.headers['content-length']) < 4096) {
      throw new Error(`${layer}: response too small to be a real map`);
    }
  }
};
Fast burn and slow burn cover different failures A fast-burn alarm evaluates the error rate over a short window and fires when the rate would exhaust the monthly error budget within about a day. It catches severe outages quickly and ignores brief spikes that consume a negligible fraction of the budget. A slow-burn alarm evaluates over a much longer window and fires at a lower rate, catching grinding degradation — a layer that fails two per cent of the time for a week — which no instantaneous threshold would ever trigger on and which nonetheless consumes the entire budget. Together they page on the failures that matter and stay quiet on the ones that do not. Fast burn short window, high multiple budget gone in about a day at this rate catches a real outage in minutes ignores a brief spike worth 1% of budget Slow burn long window, low multiple two per cent failing, for a week no instantaneous threshold would fire and it consumes the whole budget

Verification

Prove the success metric distinguishes a rendered error from a real map. Force the server to produce an error image — request a layer that does not exist with EXCEPTIONS=INIMAGE — and confirm the request is counted as a failure, not a success. If it is counted as a success, the size threshold is too low or the content-type condition is doing all the work.

Then prove the alarms can fire, using the synthetic-firing technique: publish values that push the burn rate past each threshold and confirm both alarms transition and notify. Confirm the probe fails when a layer is genuinely broken by disabling one in a staging environment. And confirm the objective is achievable by comparing the last quarter’s measured performance against it — an objective the service has never met is a permanently firing alarm, which is the same as no alarm.

Force an error image and see how it is counted Request a layer that does not exist with exceptions rendered into the image. The server returns a valid image with a success status, which is exactly the response a broken service produces in normal operation. That request must be counted as a failure by the success metric. If it is counted as a success, the body-size threshold is set too low to separate an error image from a real map tile, and the content-type condition is doing all the work — which means the metric will report full availability through the failure it was built to detect. The forced request a layer that does not exist exceptions rendered into the image a valid image, with a 200 Required accounting counted as a failure if counted as a success, the size threshold is too low to separate them Get this wrong and the metric reports full availability straight through the failure it exists to detect.

Preventing recurrence

  • Review objectives when layer classes change. A layer promoted from analytical to operational inherits a budget it was never built for.
  • Keep the probe layer list in the same module as the published layers. A new layer published without a probe is a layer with no signal, and coupling them makes the omission structural rather than a matter of memory.
  • Publish the error budget where the team sees it. A budget nobody looks at cannot inform a release decision, which is the main reason to compute one.
  • Re-derive alarm thresholds from the objective, not by hand. When the objective changes, every derived threshold should change with it in the same apply.

Frequently Asked Questions

Why is my WMS reporting perfect availability while users see errors?

Almost certainly because the monitor probes GetCapabilities, which answers from a cached document regardless of data store health, or because it checks status codes on a server configured to render exceptions inside the image. Probe GetMap with EXCEPTIONS=XML and check the body size.

Should WMS and WMTS share an objective?

No. WMTS serves pre-cut tiles and should be an order of magnitude faster than WMS rendering an arbitrary extent on demand. Sharing one objective means either an unmeetable WMS target or a WMTS target so loose it never fires.

How many synthetic probes are enough?

One per layer class plus one per business-critical layer that has low traffic. Probing every layer every minute is expensive and mostly redundant, while probing none means a rarely used layer’s failure is reported by the person who needed it.

What if the service has never met the objective?

Then the objective is wrong, the service is broken, or both — and the alarm is useless in any of those cases because it is always firing. Set the initial objective from measured performance, agree an improvement path, and tighten it deliberately rather than aspirationally.