Automating ACM Certificates for Tile CDN Endpoints

Your terraform apply has been sitting on aws_acm_certificate_validation.tiles: Still creating... for twenty minutes, the certificate shows PENDING_VALIDATION in the ACM console, and the tile CDN cutover it was supposed to unblock is now blocking a release. This is the canonical failure of automating a certificate for a CloudFront-backed tile CDN: the certificate was requested but domain control was never proven, usually because the DNS validation records did not land in the zone that actually answers for the domain, or because the certificate was created in the wrong region for CloudFront. This walkthrough is the task-level companion to TLS and Certificate Automation for Tile Endpoints within the broader Network Security & Access Control framework; it covers issuing an ACM certificate in us-east-1, wiring Route53 DNS validation so issuance completes unattended, associating the certificate with the distribution, and handling the pending-validation and renewal edge cases that keep tile maps online.

Symptom identification and triage

A stuck or failed certificate for a tile CDN surfaces in a handful of distinct shapes, and each points at a different root cause. Classify the symptom before changing any resource:

• aws_acm_certificate_validation hangs, then times out. The certificate is PENDING_VALIDATION and ACM never sees the CNAME. Either no aws_route53_record was created, it was written to the wrong hosted zone, or a name collision was silently dropped because allow_overwrite was unset. Check the record actually resolves: dig +short CNAME _x.tiles.example.com.
• Apply fails at the distribution with a region error. The specified SSL certificate ... must be in the US East (N. Virginia) region. The certificate is valid but was issued outside us-east-1; CloudFront will not attach it.
• Handshake works on some hosts, fails on others. curl -vI https://a.tiles.example.com/... returns NET::ERR_CERT_COMMON_NAME_INVALID for one shard while the apex host succeeds. The failing host is a distribution alias missing from the certificate’s SANs.
• Certificate lapses months later with no apply. A previously healthy certificate expires because the validation CNAMEs were removed in a zone refactor and ACM could no longer auto-renew.

Prerequisites and environment assumptions

This walkthrough assumes a CloudFront distribution serving raster or vector tiles from an origin (S3 or a tile server), fronted by one or more hostnames you control in Route53. To run the remediation you will need:

• Terraform >= 1.7 with the AWS provider pinned. Unpinned providers cause phantom certificate diffs and shifting validation-record shapes, so pin it explicitly:

  terraform {
    required_version = ">= 1.7.0"
    required_providers {
      aws = {
        source  = "hashicorp/aws"
        version = "~> 5.60" # acm validation + cloudfront viewer cert schema is version-sensitive
      }
    }
  }

• A us-east-1 provider alias. A CloudFront viewer certificate is only accepted from us-east-1, independent of where the distribution’s origin or the rest of your stack lives.

• A Route53 public hosted zone that is authoritative for the tile domain — its name servers must be the ones the domain’s registrar delegates to. DNS validation writes CNAMEs here; if the zone is not authoritative, ACM never sees them.

• A locking-capable state backend so a certificate swap and an alias change cannot interleave. The concurrency hazard is the same one described in State Backend Selection.

• IAM permissions for the operator: acm:RequestCertificate, acm:DescribeCertificate, route53:ChangeResourceRecordSets and route53:GetChange on the zone, and cloudfront:UpdateDistribution to attach the certificate.

Step-by-step remediation

Issue and validate the certificate first, then attach it — never point a distribution at a certificate that has not finished validating.

1. Request the certificate in us-east-1 with every tile host as a SAN. The create_before_destroy lifecycle avoids a coverage gap when the SAN set later changes and the certificate re-issues. Listing every shard and service host here is what prevents the per-host NET::ERR_CERT_COMMON_NAME_INVALID failure — the map fetches all of them and the browser validates each independently.

   provider "aws" {
     alias  = "use1"
     region = "us-east-1" # CloudFront viewer certs are region-locked to us-east-1
   }
   
   resource "aws_acm_certificate" "tiles" {
     provider                  = aws.use1
     domain_name               = "tiles.example.com"
     subject_alternative_names = [
       "a.tiles.example.com", "b.tiles.example.com",
       "c.tiles.example.com", "d.tiles.example.com",
       "wms.example.com",
     ]
     validation_method = "DNS" # DNS validation renews unattended; email validation does not
     lifecycle { create_before_destroy = true }
   }

2. Write one validation CNAME per distinct domain into the authoritative zone. The for_each over domain_validation_options produces a stable record per host; allow_overwrite prevents a re-issuance from failing on a validation name that already exists. This is the step that most often fails silently, so it is the first thing to verify when an apply hangs.

   resource "aws_route53_record" "acm_validation" {
     for_each = {
       for dvo in aws_acm_certificate.tiles.domain_validation_options :
       dvo.domain_name => {
         name   = dvo.resource_record_name
         type   = dvo.resource_record_type
         record = dvo.resource_record_value
       }
     }
     zone_id         = var.hosted_zone_id # MUST be the zone that answers for the domain
     name            = each.value.name
     type            = each.value.type
     records         = [each.value.record]
     ttl             = 60
     allow_overwrite = true
   }

3. Block on validation so nothing attaches a pending certificate. The aws_acm_certificate_validation resource waits until ACM confirms every CNAME and moves the certificate to ISSUED. Downstream resources reference its output, not the raw certificate ARN, which makes it structurally impossible to attach a still-pending certificate.

   resource "aws_acm_certificate_validation" "tiles" {
     provider                = aws.use1
     certificate_arn         = aws_acm_certificate.tiles.arn
     validation_record_fqdns = [for r in aws_route53_record.acm_validation : r.fqdn]
   }

4. Attach the validated certificate to the distribution. Reference aws_acm_certificate_validation.tiles.certificate_arn, use sni-only (required for a multi-SAN certificate), and set a modern minimum protocol. The aliases list must be a subset of the certificate SANs or the apply fails at attach time.

   resource "aws_cloudfront_distribution" "tiles" {
     enabled = true
     aliases = [
       "tiles.example.com", "a.tiles.example.com", "b.tiles.example.com",
       "c.tiles.example.com", "d.tiles.example.com", "wms.example.com",
     ]
     # ... origin, default_cache_behavior { viewer_protocol_policy = "redirect-to-https" } ...
     viewer_certificate {
       acm_certificate_arn      = aws_acm_certificate_validation.tiles.certificate_arn
       ssl_support_method       = "sni-only"
       minimum_protocol_version = "TLSv1.2_2021"
     }
   }

5. Apply and watch the validation resource resolve. If step 3 hangs, do not cancel and re-run blindly — confirm the CNAME resolves first with dig +short CNAME <validation-name>. A resolving record means ACM will pick it up within minutes; a non-resolving record means the zone is wrong or allow_overwrite dropped it.

Verification

Confirm the certificate is issued, correctly scoped, and live at the edge before closing the change.

1. Confirm ACM shows ISSUED. aws acm describe-certificate --region us-east-1 --certificate-arn <arn> --query 'Certificate.Status' must return "ISSUED", and DomainValidationOptions[].ValidationStatus must all be SUCCESS.

2. Inspect the served chain with openssl. Verify the served leaf matches the requested host and the chain is complete for a sharded host, not just the apex:

   openssl s_client -connect a.tiles.example.com:443 -servername a.tiles.example.com </dev/null 2>/dev/null \
     | openssl x509 -noout -subject -ext subjectAltName -dates

   The subjectAltName list must include a.tiles.example.com, and notAfter should be roughly 13 months out for a freshly issued ACM certificate.

3. Fetch a real tile and check the redirect and protocol. Prove HTTP is redirected and the tile serves over a modern TLS version:

   curl -vI http://tiles.example.com/10/301/384.png 2>&1 | grep -i '< location'   # expect https:// 301
   curl -vI https://tiles.example.com/10/301/384.png 2>&1 | grep -Ei 'TLSv1|SSL connection|< HTTP'

   Expect a 301 to the https:// URL, a negotiated TLSv1.2/TLSv1.3 connection, and a 200 with content-type: image/png (or application/x-protobuf for vector tiles).

4. Spot-check every SAN host. Loop the openssl/curl check across all shard and service hosts; a single missing SAN only shows up on the host it omits, so the apex passing is not sufficient.

Preventing recurrence

Encode the fix so the same certificate incident cannot recur:

• Policy-as-code gate on region and SAN coverage. Add a checkov/tfsec or CrossGuard rule that fails any certificate feeding a CloudFront viewer_certificate unless it resolves through the us-east-1 provider, and a rule asserting the distribution aliases are a subset of the certificate SANs. This blocks the two highest-frequency regressions — wrong region and a SAN gap — before merge.
• Scheduled drift and expiry detection. Run terraform plan -detailed-exitcode nightly to catch removed validation records, and add a CloudWatch alarm on the ACM DaysToExpiry metric so a stalled auto-renewal pages the owning team well before the certificate lapses.
• Own the validation records for the certificate’s whole life. Keep the aws_route53_record.acm_validation resources in the same stack as the certificate so a zone refactor cannot orphan them and disarm renewal. Treat their removal as a breaking change.
• Keep the origin private. Auto-renewal and a strict edge TLS policy only protect traffic that actually traverses the edge; keep the tile origin reachable only through CloudFront using the routing in VPC Routing for Tile Servers so nobody can bypass the certificate over plaintext.

Frequently asked questions

Related

• TLS and Certificate Automation for Tile Endpoints — the parent operational guide for certificate automation across tile endpoints
• Network Security & Access Control — the domain framework this task sits within
• CORS & CSP Configuration — the header contract, including the mixed-content directives that assume valid HTTPS
• VPC Routing for Tile Servers — keeping the tile origin private so the edge certificate cannot be bypassed