Spatial IaC Terraform & Pulumi for GIS

Pulumi IAM Policies for S3 Raster Access: Incident Response & State Recovery

Your tile servers are returning HTTP 403 AccessDenied on every s3:GetObject for a raster store that worked yesterday, and the only recent change was a Pulumi deployment touching an unrelated stack. This is the signature of an IAM policy that has drifted from its declared state — a manual console edit, a concurrent pulumi up, or a refreshed resource that silently rewrote a BucketPolicy. This guide is the incident-response companion to IAM Role Mapping for GIS and sits within the broader Network Security & Access Control framework; it covers how to isolate the denial, reconcile Pulumi state without destroying live access, and reconstruct a least-privilege policy specific to high-throughput raster delivery from Amazon S3.

Symptom Identification and Triage

A raster access failure rarely surfaces as a clean application error — it surfaces as a 403 that could originate in three different layers. Before editing any policy, classify the denial against concrete, observable signals:

  • Sustained 403 AccessDenied on tiles: every GET to tiles/{z}/{x}/{y}.png fails for one principal. Cross-reference AWS CloudTrail s3:GetObject events with S3 server access logs to capture the exact principal ARN, the evaluated condition key, and the timestamp. A consistent denial points at policy evaluation, not the network.
  • Intermittent failures across regional endpoints: some requests succeed, others fail with the same role. This is a session-context mismatch — an assumed role missing required session tags, a trust-policy boundary, or a cross-account delegation scope, all of which originate in IAM Role Mapping for GIS.
  • Browser preflight 403 with no CloudTrail entry: the request never reached S3 object evaluation. This is a CORS & CSP Configuration problem masquerading as an identity denial — confirm with the Mapbox GL JS CORS via IaC checklist before touching IAM.
  • REJECT in VPC flow logs: the packet was dropped before TLS. This is routing, not identity — route it to VPC Routing for Tile Servers rather than the policy document.

Before touching IAM, decouple identity from the network and application layers — a 403 on raster tiles almost always falls into one of three buckets:

403 AccessDenied triage: classify the denial before editing any policy A top-down decision tree. The HTTP 403 AccessDenied symptom leads to checking CloudTrail s3:GetObject events and S3 server access logs, which feeds a decision: where does the request fail? Three outcomes branch out. A REJECT in VPC flow logs is a network problem routed to VPC endpoint and routing. A denial during policy evaluation is routed to IAM principal, session tags and prefix scope. A browser preflight 403 with no CloudTrail entry is routed to CORS and CSP headers. HTTP 403 AccessDenied on raster tiles Check CloudTrail s3:GetObject events + S3 server access logs Where does it fail? Network VPC endpoint / routing IAM evaluation principal · session tags prefix scope Browser preflight CORS / CSP headers REJECT in flow logs policy evaluation no CloudTrail entry Classify the layer before editing any policy — only the centre branch is an IAM problem.

Prerequisites and Environment Assumptions

This guide assumes an AWS target with raster datasets in S3 served by a tile server or MapServer/GeoServer backend that assumes an IAM role. To execute the remediation you will need:

  • Pulumi >= 3.110 running a TypeScript program, with provider plugins pinned. Unpinned providers are a leading cause of phantom policy diffs, so pin both the SDK and the plugin:

    {
  "dependencies": {
    "@pulumi/pulumi": "3.137.0",
    "@pulumi/aws": "6.54.0"
  }
}

  • A locking-capable state backend already configured (Pulumi Cloud, or S3 + DynamoDB self-managed). The concurrency hazard that causes drift here is the same one detailed in Managing Terraform State Locks for Spatial Data — never run two pulumi up operations against one stack without a lock.

  • IAM permissions for the operator running the fix: s3:GetBucketPolicy and s3:PutBucketPolicy on the raster bucket, iam:GetRole/iam:GetRolePolicy on the consuming roles, iam:SimulatePrincipalPolicy for verification, and cloudtrail:LookupEvents for triage. State recovery also requires read/write to the state backend object and its lock table.

  • Network reachability consistent with VPC Routing for Tile Servers: the tile server should reach S3 through a Gateway VPC Endpoint, not the public internet, so the policy can scope aws:SourceVpce rather than a brittle IP range.

Step-by-Step Remediation

Reconcile state first, then rebuild the policy. Applying a new policy on top of drifted state risks a destructive replacement that revokes access for every reader at once.

  1. Reconcile drift before changing anything. Pull live attributes into the state backend, then inspect the delta. pulumi refresh makes the recorded state match reality; the preview --diff reveals whether a subsequent up would replace the BucketPolicy (which transiently denies all readers) versus update it in place.

    pulumi refresh --yes
pulumi preview --diff

  2. Surgically repair state when refresh proposes a destructive diff. If the diff shows a replacement of aws.s3.BucketPolicy, do not apply. Export the state, correct the offending resource block — a stale principal ARN, a missing condition key, malformed JSON — and re-import, so the next up is a no-op.

    pulumi stack export --file stack.json
# edit the aws:s3/bucketPolicy resource inputs in stack.json
pulumi stack import --file stack.json

  3. Rebuild the policy as least-privilege, resource-based code. Reconstruct the bucket policy with explicit principal scoping, VPC-endpoint constraint, and mandatory session tags. Each condition closes a concrete escalation path: the PrincipalTag blocks untagged roles, the prefix scope keeps the catalog role out of source rasters, and the endpoint condition removes public-internet reachability.

    S3 GetObject IAM evaluation order: explicit Deny wins, Allow needs every stage A left-to-right pipeline. A GetObject request from TileServerRole enters stage one, the Service Control Policy guardrail, then stage two, the identity-based role policy, then stage three, the resource-based bucket policy carrying the PrincipalTag and SourceVpce conditions, and finally reaches the Allow outcome. Below the pipeline, a dashed path from each stage drops to a single Deny node: an explicit Deny at any stage short-circuits to a final request Deny. Allow is only reached when all three stages permit the request. TileServerRole s3:GetObject request 1 · SCP guardrail org-wide allow boundary 2 · Identity policy role permissions (no Principal) 3 · Bucket policy resource-based PrincipalTag · SourceVpce Allow all stages permit Explicit Deny → final Deny any stage short-circuits here Evaluation order for one S3 GetObject An explicit Deny at any stage wins; Allow requires every stage to permit the request. 
    import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const current = aws.getCallerIdentityOutput();

const rasterBucket = new aws.s3.BucketV2("raster-datasets", {
    bucket: "prod-gis-raster-store",
});

new aws.s3.BucketVersioningV2("raster-versioning", {
    bucket: rasterBucket.id,
    versioningConfiguration: { status: "Enabled" },
});

// Resource-based bucket policy. "Principal" is valid in resource-based policies;
// an identity-based aws.iam.Policy must NOT declare a Principal and would be
// rejected by AWS if you tried to include one.
new aws.s3.BucketPolicy("raster-bucket-policy", {
    bucket: rasterBucket.id,
    policy: pulumi.jsonStringify({
        Version: "2012-10-17",
        Statement: [
            {
                Sid: "AllowTileServerGetObject",
                Effect: "Allow",
                Principal: { AWS: pulumi.interpolate`arn:aws:iam::${current.accountId}:role/TileServerRole` },
                Action: ["s3:GetObject", "s3:GetObjectVersion"],
                Resource: [pulumi.interpolate`${rasterBucket.arn}/*`],
                Condition: {
                    StringEquals: {
                        "aws:PrincipalTag/Environment": "production",
                        "aws:SourceVpce": "vpce-0abc123def456",
                    },
                },
            },
            {
                Sid: "AllowListForCatalog",
                Effect: "Allow",
                Principal: { AWS: pulumi.interpolate`arn:aws:iam::${current.accountId}:role/GISPlatformRole` },
                Action: "s3:ListBucket",
                Resource: rasterBucket.arn,
                Condition: {
                    StringLike: { "s3:prefix": ["tiles/*", "metadata/*"] },
                },
            },
        ],
    }),
});

  4. Apply with a clean preview. Run pulumi up only after preview --diff shows an in-place update on the policy and nothing else. For cross-account raster sharing, confirm the consuming role’s trust policy allows sts:AssumeRole with a matching tag condition before applying — the resource policy alone will not grant access if the assumed-role session never carries the Environment tag.

Verification

Confirm the fix is live at both the policy layer and the tile endpoint before closing the incident.

  1. Simulate the principal offline. Prove the role is allowed without waiting for a real request:

    aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/TileServerRole \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::prod-gis-raster-store/tiles/10/301/384.png

    The EvalDecision must be allowed. An implicitDeny means a missing condition key (usually the session tag); an explicitDeny means an SCP or competing statement still wins.

  2. Confirm in CloudTrail. After a live tile request, look up the GetObject event and verify the principal, the resolved sourceVPCEndpointId, and an absent errorCode.

  3. Probe the real endpoint. Curl a known tile through the tile server and assert a 200 with image/png, then a deliberately out-of-scope prefix (e.g. source-rasters/*) and assert it still returns 403 — proving the scope is tight, not merely open.

Preventing Recurrence

Encode the fix so the same drift cannot recur:

  • Policy-as-code gate. Add a CrossGuard (@pulumi/policy) rule to the deploy pipeline that fails any aws.s3.BucketPolicy whose statements omit an aws:PrincipalTag condition or grant s3:*. This blocks the two highest-frequency regressions — untagged access and wildcard actions — before merge.
  • Scheduled drift detection. Run pulumi preview --diff --expect-no-changes on a nightly schedule; a non-zero exit signals an out-of-band console edit and should page the owning team rather than wait for the next deploy.
  • State discipline. Keep locking enabled on the backend and never run concurrent up operations, the same way Managing Terraform State Locks for Spatial Data protects spatial state.
  • Network and lifecycle alignment. Keep all raster reads on the Gateway VPC Endpoint per VPC Routing for Tile Servers, pair the bucket with S3 lifecycle rules for GIS tiles so cold pyramids age out, and keep object ACLs disabled so the policy document is the single source of access truth.

Frequently Asked Questions

Why does my policy use Principal when an identity policy rejects it?

Because this is a resource-based bucket policy, where Principal names who the bucket grants access to. An identity-based aws.iam.Policy attached to a role must omit Principal entirely — AWS rejects an identity policy that declares one. Mixing the two shapes is a common cause of a malformed-policy apply error.

pulumi refresh wants to replace my BucketPolicy — is that safe to apply?

No. A replacement deletes the live policy before recreating it, transiently denying every reader. Export the state, correct the drifted resource inputs by hand, re-import, and confirm preview --diff shows an in-place update before running up.

The role is allowed in the policy but tiles still 403 intermittently — why?

The assumed-role session is missing the Environment tag the policy requires, so requests without it hit an implicit deny. Set session tags on sts:AssumeRole (or via the trust policy) so every session carries Environment=production, and confirm with simulate-principal-policy.

Should I scope access by IP or by VPC endpoint?

Prefer aws:SourceVpce. A aws:SourceIp CIDR breaks whenever NAT addresses rotate and still permits any host in the range; the VPC-endpoint condition ties access to a specific private path and keeps raster traffic off the public internet.