Spatial IaC Terraform & Pulumi for GIS

GeoParquet Lake Layout and Bucket Policy Design

GeoParquet turns a vector dataset into something a query engine can read selectively: columnar, compressed, with per-row-group statistics that let an engine skip whole blocks it can prove are irrelevant. That skipping only happens if the layout cooperates. A billion-row parcel dataset written as one enormous file, or as ten thousand tiny ones, or partitioned by a column nobody filters on, produces exactly the same query cost as a naive scan while looking like a modern data lake. The layout is the performance work, and the bucket policy is what keeps a lake shared across teams from becoming a lake nobody trusts. This guide extends Object Storage for Raster and Vector Data within Geospatial Resource Provisioning.

Partitioning a spatial lake

Two partition dimensions matter for geospatial data and they compose.

Time. Nearly every analytical query on an operational dataset filters by date, so year=/month= prefixes are almost always right. They are also what makes lifecycle rules and retention tractable, because a prefix maps to an age.

Space. A spatial filter is the other half of nearly every query, and the way to make it prunable is a coarse spatial key in the path. Options are an administrative code, a UTM zone, or a low-zoom tile identifier such as a quadkey at zoom 5 or 6. The quadkey is the most general because it is uniform, hierarchical and computable from any geometry without a lookup table.

The rule for depth is that a partition column must be one that queries actually filter on, and must not produce partitions so small that per-file overhead dominates. Target row groups of tens of megabytes and files of a few hundred megabytes. Below that the engine spends its time opening files; above it, row-group skipping stops helping because each group covers too much.

Crucially, partitioning does not remove the need for a bounding-box column. Write bbox — minimum and maximum coordinates as plain numeric columns — alongside the geometry, because engines can compute statistics on numeric columns and use them to skip row groups, whereas a serialised geometry column is opaque to that optimisation. The partition prunes at the file level; the bbox statistics prune at the row-group level; both are needed.

File-level and row-group-level pruning in a GeoParquet lake A query with a date range and a spatial filter prunes in two stages. First the path partition — year, month and a coarse spatial key such as a low-zoom quadkey — eliminates whole files before any of them is opened. Then, inside each surviving file, per-row-group statistics on plain numeric bounding-box columns let the engine skip blocks whose coordinate ranges cannot intersect the query envelope. Only the rows surviving both stages have their serialised geometry column read, which is the expensive part. Without the numeric bounding-box columns the second stage does not happen at all, because a serialised geometry column is opaque to statistics. Query date range + envelope Partition prune year= / month= / qk= whole files never opened Row-group skip bbox min/max statistics blocks inside a file skipped Read geometry The partition prunes files. The numeric bbox columns prune row groups. You need both. A serialised geometry column carries no usable statistics, so without bbox columns the second stage never runs. Target row groups of tens of megabytes and files of a few hundred — below that, opening files dominates; above it, each row group covers too much area for its statistics to exclude anything.

Prerequisites and environment assumptions

Terraform 1.6 or later with hashicorp/aws pinned at ~> 5.60. A writer that emits GeoParquet with the metadata the specification requires — the geo key in the file metadata carrying the CRS and geometry column name — because a Parquet file with a geometry column and no geo metadata is not GeoParquet and will be read as opaque binary by conforming readers. A catalogue if more than one team queries the lake; a table registered in a metastore is what turns a prefix convention into something discoverable.

Fix the coordinate reference system for the lake before writing anything. Mixed CRS across partitions is the defect that is hardest to unwind, because a query engine will happily union files whose coordinates mean different things and return results that are wrong rather than erroneous. One CRS per table, recorded in the file metadata, enforced by the writer.

Step-by-step layout and policy

  1. Choose the partition keys and write them into the path. Date first, spatial key second, in Hive-style key=value directories so engines discover them automatically. Keep the spatial key coarse: a zoom-5 quadkey gives roughly a thousand cells globally, which is a sensible number of partitions, whereas zoom 12 gives sixteen million and destroys the layout.

  2. Add the bbox columns at write time. Four numeric columns — bbox_xmin, bbox_ymin, bbox_xmax, bbox_ymax — derived from each geometry. They cost a little storage and they are what makes row-group skipping work.

  3. Compact on a schedule. Streaming or incremental writers produce many small files, and a lake degrades continuously without compaction. A daily job that rewrites yesterday’s partitions into properly sized files is not optional maintenance; it is part of the design.

  4. Separate write and read identities in the bucket policy. Exactly one writer identity per table prefix, and read-only access for everyone else. A lake where any consumer can write is a lake where a well-meaning notebook overwrites a partition and nobody can say when.

  5. Deny unencrypted transport and enforce the key. Standard for any bucket holding regulated geometry, and consistent with the domain-key approach in Encryption and Key Management for Spatial Data.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.60" }
  }
}

variable "writer_role_arn" { type = string }
variable "reader_role_arns" { type = list(string) }

data "aws_iam_policy_document" "lake" {
  # Exactly one writer per table prefix. A lake anyone can write to is a lake
  # where a notebook overwrites a partition and nobody can say when.
  statement {
    sid       = "SingleWriter"
    effect    = "Allow"
    actions   = ["s3:PutObject", "s3:DeleteObject"]
    resources = ["${aws_s3_bucket.lake.arn}/tables/parcels/*"]
    principals {
      type        = "AWS"
      identifiers = [var.writer_role_arn]
    }
  }

  statement {
    sid       = "BroadRead"
    effect    = "Allow"
    actions   = ["s3:GetObject", "s3:ListBucket"]
    resources = [aws_s3_bucket.lake.arn, "${aws_s3_bucket.lake.arn}/*"]
    principals {
      type        = "AWS"
      identifiers = var.reader_role_arns
    }
  }

  statement {
    sid       = "DenyPlaintextTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [aws_s3_bucket.lake.arn, "${aws_s3_bucket.lake.arn}/*"]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "lake" {
  bucket = aws_s3_bucket.lake.id

  # Scoped by prefix so it can never touch the current partitions a query
  # engine reads continuously — an unscoped rule here is the classic mistake.
  rule {
    id     = "archive-cold-partitions"
    status = "Enabled"
    filter { prefix = "tables/parcels/year=" }
    transition {
      days          = 365
      storage_class = "STANDARD_IA"
    }
  }

  # Compaction leaves behind superseded files; expiring them is what stops the
  # lake growing faster than the data it holds.
  rule {
    id     = "expire-compacted-remnants"
    status = "Enabled"
    filter { prefix = "tables/parcels/_staging/" }
    expiration { days = 7 }
  }
}

The path layout the policy above assumes, written out so the convention is unambiguous:

s3://gis-lake/tables/parcels/year=2026/month=07/qk=03201/part-0001.parquet
                             │        │         │       └── a few hundred MB per file
                             │        │         └────────── zoom-5 quadkey: ~1000 cells globally
                             │        └──────────────────── date partition: prunes and ages
                             └───────────────────────────── one writer identity owns this prefix
Small-file accumulation and the compaction job that reverses it Incremental or streaming writes append many small files into each partition. A query engine must open and read the footer of every one of them, so time is spent on file overhead rather than on data, and row-group statistics stop helping because each file holds too few rows to have meaningful groups. A scheduled compaction job rewrites the previous day's partitions into files of a few hundred megabytes with row groups of tens of megabytes, restoring both levels of pruning. The superseded files are moved to a staging prefix and expired after a short retention, which keeps the lake from growing faster than the data it holds. Before compaction many tiny files footers dominate the read daily compaction of yesterday's partitions After compaction part-0001 part-0002 Superseded files moved to _staging/ and expired after seven days without this the lake grows faster than its data

Verification

Run a representative query with the engine’s explain output and confirm partitions were pruned — the number of files scanned should be a small fraction of the total, and if it equals the total the partition keys do not match the filter. Then confirm row-group skipping by comparing bytes scanned against total file size for a spatially narrow query; a query over one city that scans the whole national file has bbox columns missing or unwritten.

Confirm the files are genuine GeoParquet by reading the geo metadata key and checking the declared CRS matches what you intended. Check file and row-group sizes across a few partitions against the targets. And test the policy: attempt a write as a reader identity and require it to fail, because a lake’s single-writer guarantee is worth exactly as much as the test that proves it.

Two ratios in the explain output, two different faults Files scanned divided by files present measures partition pruning. If the ratio is close to one, the partition keys do not match the filter the query actually uses, and no amount of statistics inside the files will help. Bytes scanned divided by the total size of the files that were opened measures row-group skipping. If that ratio is close to one on a spatially narrow query, the numeric bounding-box columns are missing or were never written, so the engine had no statistics to exclude anything with. The two ratios fail independently and have different remedies, which is why reading them separately matters. files scanned / files present measures partition pruning near one means the partition keys do not match the filter used statistics cannot rescue this bytes scanned / bytes opened measures row-group skipping near one on a narrow query means the bbox columns are missing partitioning cannot rescue this The two fail independently and have different remedies — which is the reason to read them separately.

Preventing recurrence

  • Monitor file count per partition. A rising count is the earliest signal that compaction has stopped, and it precedes the query slowdown by days.
  • Assert the CRS at write time. A writer that refuses to emit a file whose CRS differs from the table’s declared one prevents the defect that is hardest to unwind.
  • Scope every lifecycle rule by prefix. An unscoped archival transition will eventually move a hot partition into a class with retrieval latency, which presents as an inexplicably slow query.
  • Keep the catalogue and the layout in one module. A table registered in a metastore whose partition scheme has drifted from the actual prefixes produces queries that silently miss data.

Frequently Asked Questions

Do I need bbox columns if I already partition spatially?

Yes. The partition prunes at the file level and the bbox statistics prune at the row-group level, and a coarse partition still leaves a lot of data inside each file. Without numeric bbox columns the engine cannot skip anything inside a file, because a serialised geometry column carries no usable statistics.

What zoom level should the quadkey partition use?

Coarse — zoom 5 or 6, giving roughly a thousand to four thousand cells globally. The aim is enough partitions to prune meaningfully and few enough that each holds properly sized files. A fine quadkey produces millions of nearly empty partitions and is strictly worse than no spatial partition at all.

Should GeoParquet replace PostGIS?

They answer different questions. GeoParquet is for analytical scans over large historical volumes; PostGIS is for transactional access, indexed point queries and the live rendering path described in Vector Tile Service Provisioning. Most platforms run both, with the lake fed from the database.

How do I handle late-arriving data in a date partition?

Write it into the partition it belongs to and let the next compaction pass fold it in. Re-partitioning by arrival time to avoid rewriting is tempting and it breaks every query that filters on event date, which is all of them.