Spatial IaC Terraform & Pulumi for GIS

Mutual TLS Between Tile Services and Upstream APIs

A tile platform is rarely self-contained. The renderer calls a geocoding service, an elevation API, a partner’s feature service, a licensing check. Each of those calls carries a credential, and a static API key in an environment variable is a credential that leaks in logs, survives in images, and cannot be revoked for one caller without revoking it for all. Mutual TLS replaces it with a certificate that proves who the caller is at the transport layer, cannot be replayed from a log line, and expires whether or not anyone remembers to rotate it. This guide extends TLS Certificate Automation within Network Security and Access Control.

What mutual TLS actually changes

In ordinary TLS the client verifies the server. In mutual TLS both directions verify, so the server also demands a certificate from the client and validates it against a certificate authority it trusts. Three consequences follow, and the third is the one that decides whether this is worth the effort.

The credential cannot be copied out of a log. A bearer token appears in headers, proxies and traces; a private key never travels on the wire, so an attacker with a full request capture still cannot impersonate the client.

Revocation is per identity and immediate. Each caller has its own certificate. Removing one caller is revoking one certificate, not rotating a shared secret and redeploying every consumer of it.

Expiry is enforced by the protocol. This is the double edge. A certificate that expires stops working, in production, at a moment determined months earlier by someone who has probably moved on. Mutual TLS without automated renewal is a scheduled outage, and the automation is not optional in the way it is for a server certificate that at least produces a browser warning first.

What a client certificate replaces An API key travels in every request, so it appears in application logs, proxy logs and distributed traces, and anyone who captures one can replay it. Revoking it means rotating a shared secret and redeploying every consumer. A client certificate instead proves possession of a private key that never travels on the wire, so a full request capture does not yield a usable credential. Each caller has its own certificate, so revocation is per identity and immediate. The trade is that expiry is enforced by the protocol: an unrenewed certificate stops working in production at a moment set months earlier, which makes automated renewal a hard requirement rather than good practice. API key travels in every request lands in logs, proxies, traces replayable by anyone who sees it revoking it redeploys every consumer never expires unless someone acts Client certificate the private key never travels a full capture yields nothing usable revocation is per caller, immediate identity is verified at the transport layer expires on schedule — renewal must be automated

Prerequisites and environment assumptions

Terraform 1.6 or later with hashicorp/aws pinned at ~> 5.60. A private certificate authority — a managed one, or an internal authority whose issuance you can automate; a manually operated authority reintroduces exactly the human step this design removes. Somewhere to store issued certificates and keys that the workload can read at start-up, which is a secrets manager rather than a container image. And an upstream service that supports client certificate validation; if the upstream is a third party that only offers API keys, mutual TLS is not available for that hop and the honest answer is to say so rather than to build something that looks like it.

Decide the certificate lifetime deliberately. Short lifetimes — days rather than years — are better security and demand more reliable automation. The workable rule is that lifetime should be several times the renewal interval, so a single failed renewal is a warning rather than an outage, and monitoring should alert on time-to-expiry rather than on renewal failure alone.

Step-by-step implementation

  1. Create a private certificate authority for internal service identity. It should be distinct from whatever issues your public server certificates, because the trust decisions are different: this authority’s certificates say “this is a service in our platform”, and nothing outside the platform should trust it.

  2. Issue one certificate per calling service, not per environment. The subject common name should identify the service — tile-renderer.prod — because that string is what the upstream will authorise against, and a certificate shared across services makes per-caller revocation impossible.

  3. Store the certificate and key in a secrets manager and mount at start-up. The private key must not be in the image, in the task definition, or in an environment variable that appears in a console. Fetching at start-up also means a rotated certificate is picked up on the next deployment or restart.

  4. Configure the upstream to require and authorise the client certificate. Requiring a certificate is authentication; checking which certificate is authorisation, and only the second one distinguishes an allowed caller from any other holder of a certificate from the same authority. Match on the subject or on a certificate attribute, and treat “signed by our authority” alone as insufficient.

  5. Automate renewal with headroom and alert on expiry. A renewal job that runs at one third of the certificate lifetime gives two failures’ worth of slack. The alert must be on remaining validity, because a renewal job that has silently stopped running reports no failures at all.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.60" }
  }
}

resource "aws_acmpca_certificate_authority" "internal" {
  type = "ROOT"
  certificate_authority_configuration {
    key_algorithm     = "RSA_2048"
    signing_algorithm = "SHA256WITHRSA"
    subject {
      common_name  = "Spatial Platform Internal CA"
      organization = "Example Mapping"
    }
  }
  # Distinct from whatever issues public server certificates: this authority
  # says "a service inside our platform", and nothing outside should trust it.
  usage_mode = "SHORT_LIVED_CERTIFICATE"
}

# One certificate per CALLING SERVICE. Sharing one across services makes
# per-caller revocation impossible, which is most of the point.
resource "aws_acm_certificate" "tile_renderer_client" {
  domain_name               = "tile-renderer.prod.internal"
  certificate_authority_arn = aws_acmpca_certificate_authority.internal.arn
  key_algorithm             = "RSA_2048"

  options {
    certificate_transparency_logging_preference = "DISABLED"
  }

  lifecycle {
    create_before_destroy = true
  }
}

# The private key is fetched at start-up, never baked into an image and never
# placed in an environment variable that a console will render.
resource "aws_secretsmanager_secret" "client_cert" {
  name = "tls/client/tile-renderer"
  tags = {
    Service  = "tile-renderer"
    Rotation = "automated"
  }
}

# Alert on REMAINING VALIDITY, not on renewal failure: a renewal job that has
# silently stopped running produces no failures at all.
resource "aws_cloudwatch_metric_alarm" "client_cert_expiry" {
  alarm_name          = "tile-renderer-client-cert-expiry"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 1
  threshold           = 10 # days
  metric_name         = "DaysToExpiry"
  namespace           = "AWS/CertificateManager"
  period              = 86400
  statistic           = "Minimum"
  dimensions          = { CertificateArn = aws_acm_certificate.tile_renderer_client.arn }
  alarm_actions       = [var.oncall_topic_arn]
}

variable "oncall_topic_arn" { type = string }
# Upstream side. Requiring a certificate is AUTHENTICATION; checking which
# certificate is AUTHORISATION, and only the second distinguishes an allowed
# caller from any other holder of a certificate from the same authority.
server {
    listen 443 ssl;
    server_name elevation-api.internal;

    ssl_certificate     /etc/ssl/server.crt;
    ssl_certificate_key /etc/ssl/server.key;

    ssl_client_certificate /etc/ssl/internal-ca.crt;
    ssl_verify_client      on;
    ssl_verify_depth       2;

    location / {
        # "Signed by our CA" is not enough — every service in the platform has
        # one of those. Authorise the specific subject.
        if ($ssl_client_s_dn !~ "CN=tile-renderer\.prod\.internal") {
            return 403;
        }
        proxy_set_header X-Client-Identity $ssl_client_s_dn;
        proxy_pass http://elevation_backend;
    }
}
Renewal headroom and why the alarm watches validity A certificate with a thirty-day lifetime is renewed on a ten-day cycle. A single failed renewal leaves twenty days of validity remaining, which is a warning rather than an incident, and two consecutive failures still leave ten days. The alarm is configured on remaining validity rather than on renewal job failure, because the failure mode that actually causes outages is a renewal job that has silently stopped running altogether — it produces no failure events, so an alarm watching for failures sees nothing while the certificate quietly approaches expiry. issued day 10 · renew day 20 · renew day 30 · expiry first renewal window one failure survivable two failures — act now The alarm watches remaining validity, not renewal failures — a job that has stopped running reports no failures.

Verification

Prove both halves separately. Connect from the renderer with its certificate and confirm the call succeeds. Then connect without a certificate and confirm the upstream refuses — that proves ssl_verify_client is actually on rather than configured and not reloaded. Then, most importantly, connect with a different valid certificate from the same authority and confirm the upstream refuses it too. That third test is the one that distinguishes authentication from authorisation, and it is the one most often skipped.

Confirm the private key is not reachable anywhere it should not be: not in the image layers, not in the task definition, not in an environment variable listed by the console. Finally, confirm the renewal automation works by observing a real renewal rather than trusting the schedule — the first renewal is where a misconfigured issuance permission shows up, and it is far better to see it while the old certificate has three weeks left.

The third test is the one that is usually skipped Three connection attempts prove different things. Connecting with the renderer's own certificate must succeed, which shows the happy path works. Connecting with no certificate must be refused, which shows client verification is actually enabled rather than configured and never reloaded. Connecting with a different but entirely valid certificate issued by the same authority must also be refused — and that is the test almost always skipped, because it is the only one that distinguishes authenticating a caller from authorising a specific one. Without it, every service in the platform can call every upstream. Own certificate must succeed the happy path works everyone tests this one No certificate must be refused verification is genuinely on not merely configured A different valid one must also be refused authorisation, not authentication and it is the one skipped Skip the third and every service in the platform can call every upstream, because they all hold a valid certificate.

Preventing recurrence

  • Alarm on days remaining, per certificate. The certificate that expires is always the one nobody had an alarm on, and a per-certificate alarm derived from the module means new services get one automatically.
  • Keep issuance in code. A certificate issued by hand during an incident has no renewal automation attached and will expire silently in a year.
  • Record the authorised subject alongside the upstream configuration. The authorisation rule is a string match, and a service rename that changes the subject breaks the call in a way whose error message points at TLS rather than at the rename.
  • Rehearse revocation. Revoking one caller’s certificate and confirming that only that caller stops working is the test that proves the per-identity property you adopted mutual TLS for.

Frequently Asked Questions

Is a certificate signed by our authority enough to authorise a caller?

No, and treating it as enough is the most common mistake in a mutual TLS deployment. Every service in the platform holds such a certificate, so validation alone lets any of them call any upstream. Authorise on the subject or on a specific certificate attribute in addition to validating the chain.

What lifetime should client certificates have?

Short, provided renewal is automated — days to weeks rather than years. The rule that matters is that the lifetime should be several times the renewal interval, so one failed renewal is a warning and not an outage.

Can I use mutual TLS with a third-party API?

Only if they support it. Many do not, and the honest response is to keep an API key for that hop while confining it: store it in the secrets manager, rotate it on a schedule as described in Rotating PostGIS Credentials in Terraform Without Downtime, and keep it out of logs.

Does mutual TLS replace network controls?

No. It authenticates the caller; it does not stop an unauthorised host from reaching the endpoint and consuming resources trying. Keep the security group and routing controls in place — the layers answer different questions and both are cheap.