Spatial IaC Terraform & Pulumi for GIS

Provisioning CloudWatch Dashboards for Tile Latency

Most tile platform dashboards are built for the wrong reader. They show aggregate latency, total requests and a CPU graph, which is the view you want when writing a monthly report and useless at 3am when a page has fired and you have four minutes before someone asks what is happening. A dashboard for a tile platform should answer, in order: is it the cache, is it the database, or is it one layer — and it should answer each in a single glance without anyone typing a query. Provisioning it from the same module as the service is what makes it exist the moment the service does. This guide extends Observability for Spatial Infrastructure within CI/CD Automation and Governance.

Designing for the questions, not the metrics

The failure of most dashboards is that they show what is easy to plot. Design instead from the diagnostic sequence a responder actually follows on a tile platform, which is remarkably consistent.

Is the cache still absorbing traffic? The hit ratio is the single most informative number, because a fall from 95 to 80 per cent quadruples origin load with no change in user traffic. This panel goes top-left, where the eye lands first.

Is the origin saturated, and on what? Connection pool utilisation against capacity, not renderer CPU — the pool is the ceiling that binds first, and it fails as a cliff rather than a slope, so it must be visible before it is reached.

Is it one layer or all of them? Latency broken down by layer and zoom band. An aggregate that hides a single pathological layer is the reason a responder ends up querying logs by hand.

Is it the database? Read latency and active connections at the cluster, alongside the top statements by total time. A slow tile query is usually a slow spatial query.

Did something change? A deployment annotation overlay. Half of all incidents correlate with a change, and a dashboard that shows deployments answers that question before anyone has to ask in a channel.

Panel layout following the diagnostic sequence The dashboard is arranged in the order a responder asks questions rather than by metric source. Top-left, where the eye lands first, is the cache hit ratio, because a fall in it quadruples origin load without any change in user traffic. Beside it is connection pool utilisation against capacity, which is the ceiling that binds before renderer CPU and which fails as a cliff rather than a slope. The middle row breaks latency down by layer and zoom band, so a single pathological layer is visible rather than hidden inside an aggregate. Below that are database read latency and the top statements by total time, since a slow tile query is usually a slow spatial query. A deployment annotation overlay runs across the full width, because roughly half of incidents correlate with a change. 1 · Cache hit ratio the most informative number on the page 95% → 80% quadruples origin load 2 · Pool utilisation vs capacity the ceiling that binds first a cliff, not a slope — see it early 3 · Latency by layer and zoom band one pathological layer is invisible in an aggregate — and it is usually one layer 4 · Database read latency plus top statements by total time 5 · Deployments overlay half of incidents correlate with a change

Prerequisites and environment assumptions

Terraform 1.6 or later with hashicorp/aws pinned at ~> 5.60. Metrics emitted with the bounded dimensions from Observability for Spatial Infrastructure — zoom band, layer, cache result — because every panel below depends on them and none of it is possible against aggregates. A deployment event stream to annotate against.

Decide the time range and refresh before building. A dashboard defaulting to a week is a reporting dashboard; one defaulting to three hours is an incident dashboard. If you need both, build both — a single dashboard tuned for two purposes serves neither, and duplicating a panel is cheap while a misread graph at 3am is not.

Step-by-step implementation

  1. Compute derived values in metric expressions, not in the reader’s head. Hit ratio as a percentage, pool utilisation as a percentage of capacity. A panel showing hits and misses as two lines requires arithmetic under pressure, and the arithmetic gets done wrong.

  2. Break latency down by the dimensions that matter and cap the series count. Layer and zoom band, with the top few by latency rather than every combination — a panel with forty lines communicates nothing.

  3. Put the database beside the tile tier, not on a separate dashboard. The whole point is answering “is it the database” without navigating, and a separate dashboard adds a navigation step at the worst moment.

  4. Add annotations for deployments and for alarm state. Alarm annotations mean the responder sees which alarm fired and when, in the same view as the metrics.

  5. Generate the dashboard from the same module as the service. A dashboard provisioned alongside the resource exists the moment the resource does, and a deleted panel is drift rather than an unnoticed gap.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.60" }
  }
}

variable "service_name" { type = string }
variable "cluster_id" { type = string }
variable "distribution_id" { type = string }

resource "aws_cloudwatch_dashboard" "tiles" {
  dashboard_name = "${var.service_name}-tiles"

  dashboard_body = jsonencode({
    # Three hours: this is an incident dashboard. Reporting views get their own.
    start          = "-PT3H"
    periodOverride = "auto"

    widgets = [
      {
        type = "metric", x = 0, y = 0, width = 12, height = 6
        properties = {
          title = "Cache hit ratio — the first question"
          # Derived here, not in the reader's head. Two raw lines would require
          # arithmetic under pressure, and the arithmetic gets done wrong.
          metrics = [
            [{ expression = "100 * hits / IF(requests > 0, requests, 1)", label = "hit ratio %", id = "ratio" }],
            ["AWS/CloudFront", "CacheHitRate", "DistributionId", var.distribution_id, { id = "hits", visible = false }],
            ["AWS/CloudFront", "Requests", "DistributionId", var.distribution_id, { id = "requests", visible = false }],
          ]
          yAxis  = { left = { min = 0, max = 100 } }
          annotations = {
            horizontal = [{ label = "origin load doubles below here", value = 90 }]
          }
          view = "timeSeries", stat = "Sum", region = data.aws_region.current.name
        }
      },
      {
        type = "metric", x = 12, y = 0, width = 12, height = 6
        properties = {
          title = "Connection pool utilisation — the ceiling that binds first"
          metrics = [
            [{ expression = "100 * active / IF(capacity > 0, capacity, 1)", label = "pool used %", id = "util" }],
            ["SpatialPlatform/Tiles", "ActiveConnections", { id = "active", visible = false }],
            ["SpatialPlatform/Tiles", "PoolCapacity", { id = "capacity", visible = false }],
          ]
          # Saturation is a cliff. The line must be visible before it is reached.
          annotations = { horizontal = [{ label = "saturation", value = 90, fill = "above" }] }
          yAxis = { left = { min = 0, max = 100 } }
          view  = "timeSeries", region = data.aws_region.current.name
        }
      },
      {
        type = "metric", x = 0, y = 6, width = 24, height = 6
        properties = {
          title = "p99 origin latency by zoom band — is it one layer, or all of them?"
          metrics = [
            ["SpatialPlatform/Tiles", "OriginLatency", "ZoomBand", "z0-z8", { label = "z0-z8" }],
            ["...", "ZoomBand", "z9-z12", { label = "z9-z12" }],
            ["...", "ZoomBand", "z13-z16", { label = "z13-z16 (dense)" }],
            ["...", "ZoomBand", "z17+", { label = "z17+" }],
          ]
          stat = "p99", view = "timeSeries", region = data.aws_region.current.name
        }
      },
      {
        type = "metric", x = 0, y = 12, width = 12, height = 6
        properties = {
          # Beside the tile tier deliberately: "is it the database" must be
          # answerable without navigating away.
          title = "Database — read latency and active connections"
          metrics = [
            ["AWS/RDS", "ReadLatency", "DBClusterIdentifier", var.cluster_id],
            ["AWS/RDS", "DatabaseConnections", "DBClusterIdentifier", var.cluster_id, { yAxis = "right" }],
          ]
          view = "timeSeries", region = data.aws_region.current.name
        }
      },
      {
        type = "log", x = 12, y = 12, width = 12, height = 6
        properties = {
          title = "Slowest tile queries in the window"
          query = join(" | ", [
            "SOURCE '/aws/rds/cluster/${var.cluster_id}/postgresql'",
            "filter @message like /ST_AsMVT/",
            "stats avg(duration_ms) as avg_ms, count(*) as n by query_fingerprint",
            "sort avg_ms desc",
            "limit 10",
          ])
          view = "table", region = data.aws_region.current.name
        }
      },
    ]
  })
}

data "aws_region" "current" {}
Build two dashboards, not one compromise An incident dashboard defaults to a three-hour window, shows derived ratios rather than raw counters, carries annotated threshold lines so a value can be judged at a glance, and places the database panels beside the tile tier so the responder never navigates. A reporting dashboard defaults to a month, shows totals, trends and cost alongside traffic, and is read deliberately rather than under pressure. A single dashboard tuned for both ends up with a time range wrong for one of them and a panel density wrong for the other, so duplicating a few panels is the cheaper trade. Incident dashboard defaults to three hours ratios computed, not raw counters threshold lines drawn on every panel database beside the tile tier read in four minutes, under pressure Reporting dashboard defaults to a month totals, trends, cost beside traffic density is fine — it is read deliberately answers "how are we doing" never opened during an incident One dashboard tuned for both has the wrong time range for one reader and the wrong density for the other.

Verification

Verify the dashboard the way it will be used. Open it cold, with no context, and try to answer the three questions in order — cache, saturation, layer. If any of them requires typing a query, changing a time range or navigating to another page, the layout has not done its job and the fix is a panel rather than a runbook paragraph.

Then verify the panels are actually populated. An expression referencing a metric that is never published renders as an empty graph, which reads as “nothing is happening” rather than as “this panel is broken” — and that misreading during an incident is worse than having no panel. Check each panel returns data over a period when the service was busy, and check the log-insights widget’s query against a real log group rather than trusting the field names.

Finally, verify it after a deployment. The annotation overlay should show the deployment, and the panels should show whatever the deployment did. A dashboard that cannot show the effect of a known change will not show the effect of an unknown one.

An empty panel and a quiet service look the same A panel whose expression references a metric that is never published renders as an empty graph. During an incident that reads as nothing is happening here, which is a far more damaging misreading than an obviously broken panel would be. Every panel must therefore be confirmed to return data over a period when the service was genuinely busy, and every log query must be run against a real log group rather than trusting the field names to be right. The two failures are indistinguishable on the page, and only one of them is true. Panel is broken the metric is never published renders as an empty graph Service is quiet genuinely no traffic renders as an empty graph Identical on the page, and only one is true — so confirm every panel over a period the service was busy.

Preventing recurrence

  • Provision dashboards in the same module as the service. A service deployed without its dashboard runs unmonitored until someone notices, and coupling them removes the window entirely.
  • Treat a deleted panel as drift. The drift detection in Drift Detection and Remediation should cover dashboards, which are routinely excluded because they are not thought of as infrastructure.
  • Add a panel after every incident that needed an ad-hoc query. That query is a missing panel, and adding it is how the dashboard improves rather than merely accumulating.
  • Prune panels nobody looks at. Density has a cost at 3am, and a panel that has never informed a decision is taking space from one that would.

Frequently Asked Questions

Why not put everything on one dashboard?

Because the reader differs. An incident reader has four minutes and needs three questions answered in order; a reporting reader has an hour and wants a month of trend. A single view compromises the time range and the density for both, and duplicating a few panels is far cheaper than a graph misread under pressure.

Should the dashboard show renderer CPU?

Low on the page, if at all. CPU is the last ceiling to bind on a tile platform and the first thing people look at, which is precisely the misdirection the layout exists to correct. Pool utilisation and cache hit ratio are the numbers that predict a problem.

How many series should a latency panel show?

Four to six. Zoom bands work well because they are few and meaningful. A panel with one line per layer becomes unreadable at a dozen layers, so show the top few by latency and put the full breakdown in a linked query.

Do annotations really matter?

They are among the highest-value elements on the page. Roughly half of incidents correlate with a change, and a deployment annotation answers “did something change” before anyone has to ask in a channel and wait for a reply.