Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 117 additions & 0 deletions infra/gcp/terraform/k8s-infra-oci-proxy-prod/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
## AWS ↔ GCP region pairing and mapping

This document contains the GCP regions that we serve image registries from including future regions.

At a high level:
- A global GCP loadbalancer routes traffic to the closest Cloud Run service
- Image Manifests are fetched from GCP
- Traffic originating from GCP fetches image blobs from GCP Artifact Registry
- Traffic originating from AWS fetches image blobs from S3 Buckets
- Traffic originating from outside of GCP and AWS is fetched from S3 buckets
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it from S3 buckets directly or from CloudFront?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Directly from S3 right now for AWS clients

- If a GCP region doesn't have an paired AWS region, the user fetches image blobs from AWS Cloudfront CDN.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this at all. Are we talking about the case where a GCP region doesn't have an active GCP image registry, and it doesn't have a paired AWS region? Because otherwise, GCP regions should always pull from GCP.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since Blobs served from GCP to non cloud users? is currently No, so blobs come from AWS.

Example:
An office worker in London fetching an image gets the blobs from an AWS S3 bucket in London
An office worker in Helsinki fetching an image receives the blobs from AWS Cloudfront, as there are no AWS regions in Helsinki/Finland, and the nearest S3 bucket is in Stockholm/Frankfurt. You'll get better response times from Cloudfront edge in Helsinki(AWS has more PoPs/edges than regions)

In the future, we want all non-AWS/GCP clients to fetch from Cloudfront.


| Metro / Country | AWS region | GCP region | Is the GCP region deployed? | Blobs served from GCP to non cloud users? | Active GCP Image Registry | Active S3 Bucket | Nearest Blob Location
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blobs served from GCP to non cloud users? is this column ever Yes? It appears that all values are No, in which case I'm wondering do we really need it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to do that in the future, but right now it is not possible.

|---|---|---|---|---|---|---|---|
| South Africa | `af-south-1` | `africa-south1` | 🔴 | No | No | Yes | Same Region | Same Region |
| Taiwan | `ap-east-2` | `asia-east1` | 🟢 | No | Yes | Yes | Same Region |
| Hong Kong (SAR) | `ap-east-1` | `asia-east2` | 🔴 | No | No | Yes | Same Region | Same Region |
| Tokyo, Japan | `ap-northeast-1` | `asia-northeast1` | 🟢 | No | Yes | Yes | Same Region |
| Osaka, Japan | `ap-northeast-3` | `asia-northeast2` | 🟢 | No | Yes | Yes | Same Region |
| Seoul, South Korea | `ap-northeast-2` | `asia-northeast3` | 🔴 | No | No | Yes | Same Region |
| Mumbai, India | `ap-south-1` | `asia-south1` | 🟢 | No | Yes | Yes | Same Region |
| Hydrebad, India | — | `asia-south2` | 🔴 | No | No | — | AWS Cloudfront |
| Singapore | `ap-southeast-1` | `asia-southeast1` | 🔴 | No | Yes | Yes | Same Region |
| Jakarta, Indonesia | `ap-southeast-3` | `asia-southeast2` | 🔴 | No | Yes | Yes | Same Region |
| Sydney, Australia | `ap-southeast-2` | `australia-southeast1`| 🟢 | No | Yes | Yes | Same Region |
| Melbourne, Australia | `ap-southeast-4` | `australia-southeast2` | 🔴 | No | No | Yes | Same Region |
| Warsaw, Poland | — | `europe-central2` | 🔴 | No | No | — | AWS Cloudfront |
| Hamina, Finland | — | `europe-north1` | 🟢 | No | Yes | — | AWS Cloudfront |
| Stockholm, Sweden | `eu-north-1` | `europe-north2` | 🔴 | No | No | Yes | Same Region |
| Madrid, Spain | `eu-south-2` | `europe-southwest1` | 🟢 | No | Yes | Yes | Same Region |
| St. Ghislain, Belgium | — | `europe-west1` | 🟢 | No | Yes | — | Europe |
| London, UK | `eu-west-2` | `europe-west2` | 🟢 | No | Yes | Yes | Same Region |
| Frankfurt, Germany | `eu-central-1` | `europe-west3` | 🟢 | No | Yes | Yes | Same Region |
| Eemshaven, Netherlands | — | `europe-west4` | 🟢 | No | Yes | Yes | Europe |
| Zürich, Switzerland | `eu-central-2` | `europe-west6` | 🔴 | No | No | Yes | Same Region |
| Milan, Italy | `eu-south-1` | `europe-west8` | 🟢 | No | Yes | Yes | Same Region |
| Paris, France | `eu-south-2` | `europe-west9` | 🟢 | No | Yes | Yes | Same Region |
| Berlin, Germany | — | `europe-west10` | 🟢 | No | Yes | Yes | Same Country |
| Turin, Italy | — | `europe-west12` | 🔴 | No | No | — | AWS Cloudfront |
| Doha, Qatar | — | `me-central1` | 🔴 | No | No | — | AWS Cloudfront |
| Dammam, Saudi Arabia | — | `me-central2` | 🔴 | No | No | — | AWS Cloudfront |
| Tel Aviv, Israel | `il-central-1` | `me-west1` | 🔴 | No | No | Yes | Europe |
| Montréal, Canada | `ca-central-1` | `northamerica-northeast1` | 🔴 | No | No | Yes | Same Region |
| Toronto, Canada | — | `northamerica-northeast2` | 🔴 | No | No | — | AWS Cloudfront |
| Querétaro, Mexico | `mx-central-1` | `northamerica-south1` | 🔴 | No | No | Yes | Same Region |
| São Paulo, Brazil | `sa-east-1` | `southamerica-east1` | 🔴 | No | No | Yes | Same Region |
| Santiago, Chile | — | `southamerica-west1` | 🟢 | No | Yes | No | AWS Cloudfront |
| Council Bluffs (Iowa), USA | — | `us-central1` | 🟢 | No | Yes | Yes | `us-east-2` |
| Moncks Corner (South Carolina), USA | — | `us-east1` | 🟢 | No | Yes | — | `us-east-1` |
| Ashburn (N. Virginia), USA | `us-east-1` | `us-east4` | 🟢 | No | Yes | Yes | Same Region |
| Columbus (Ohio), USA | `us-east-2` | `us-east5` | 🟢 | No | Yes | Yes | Same Region |
| Dallas (Texas), USA | — | `us-south1` | 🟢 | No | Yes | — | `us-east-2` |
| The Dalles (Oregon), USA | `us-west-2` | `us-west1` | 🟢 | No | Yes | Yes | Same Region |
| California, USA | `us-west-1` | `us-west2` | 🟢 | No | Yes | Yes | Same Region |
| Salt Lake City (Utah), USA | — | `us-west3` | 🔴 | No | No | — | AWS Cloudfront |
| Las Vegas (Nevada), USA | — | `us-west4` | 🔴 | No | No | — | AWS Cloudfront |


Priority Regions:
- P1
- `af-south-1`. 1st region in Africa
- `me-central1`. 1st region in the Middle East
- `asia-east2`. A paired region that will also serve AWS China traffic.
- `northamerica-northeast1`. First paired region in Canada
- P2
- `europe-north2`. A paired region
- `northamerica-south1` A paired region
- `asia-southeast1`. A paired region
- Backlog
- Remaining US regions

Regions we should replace given promoter capacity:
- `asia-northeast2`, another region of the same country is already active
- `europe-west10`, another region of the same country is already active

As of 13th of December 2025, all the AWS regions that publicly available have been populated and configured in archeio.

```
# aws ec2 describe-regions --all-regions --query "Regions[].RegionName" --output json | jq .[] | awk '{print $0","}' | sort --version-sort
"af-south-1",
"ap-east-1",
"ap-east-2",
"ap-northeast-1",
"ap-northeast-2",
"ap-northeast-3",
"ap-southeast-1",
"ap-southeast-2",
"ap-southeast-3",
"ap-southeast-4",
"ap-southeast-5",
"ap-southeast-6",
"ap-southeast-7",
"ap-south-1",
"ap-south-2",
"ca-central-1",
"ca-west-1",
"eu-central-1",
"eu-central-2",
"eu-north-1",
"eu-south-1",
"eu-south-2",
"eu-west-1",
"eu-west-2",
"eu-west-3",
"il-central-1",
"me-central-1",
"me-south-1",
"mx-central-1",
"sa-east-1",
"us-east-1",
"us-east-2",
"us-west-1",
"us-west-2",
```

Helpful Guides:
- https://cloudregionsmap.z6.web.core.windows.net/
5 changes: 2 additions & 3 deletions infra/gcp/terraform/k8s-infra-oci-proxy-prod/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,8 @@ locals {
module "oci-proxy" {
source = "../modules/oci-proxy"
// ***** production vs staging variables inputs *****
//
// explicitly using default digest here vs staging which overrides it
digest = null
// v0.6.0
digest = "sha256:1f004a487c09c76ebf14d1dc01434aa17e31e39def56451ec5727160fdcd0017"
domain = "registry.k8s.io"
project_id = local.project_id
service_account_name = "oci-proxy-prod"
Expand Down
13 changes: 7 additions & 6 deletions infra/gcp/terraform/modules/oci-proxy/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ locals {
environment_variables = [
{
name = "DEFAULT_AWS_BASE_URL",
// AWS Cloudfront
value = "https://d39mqg4b1dx9z1.cloudfront.net",
// AWS ap-east-2 is Taiwan
value = "https://prod-registry-k8s-io-ap-east-2.s3.dualstack.ap-east-2.amazonaws.com",
},
{
name = "UPSTREAM_REGISTRY_ENDPOINT",
Expand Down Expand Up @@ -111,8 +111,8 @@ locals {
environment_variables = [
{
name = "DEFAULT_AWS_BASE_URL",
// AWS eu-central-1 is Frankfurt
value = "https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com",
// AWS Cloudfront
value = "https://d39mqg4b1dx9z1.cloudfront.net",
},
{
name = "UPSTREAM_REGISTRY_ENDPOINT",
Expand Down Expand Up @@ -183,8 +183,9 @@ locals {
environment_variables = [
{
name = "DEFAULT_AWS_BASE_URL",
// AWS eu-west-1 is Ireland
value = "https://prod-registry-k8s-io-eu-west-1.s3.dualstack.eu-west-1.amazonaws.com",
// AWS eu-west-2 is London
// https://github.com/kubernetes/registry.k8s.io/blob/main/cmd/archeio/internal/app/buckets.go#L111
value = "https://767373bbdcb8270361b96548387bf2a9ad0d48758c35-eu-west-2.s3.dualstack.eu-west-2.amazonaws.com",
},
{
name = "UPSTREAM_REGISTRY_ENDPOINT",
Expand Down
1 change: 1 addition & 0 deletions registry.k8s.io/images/k8s-staging-infra-tools/images.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"sha256:a8e39369650944a3bc7f7ce5bc6f28dbc79b4eb856e54d3968e84f0485a29019": ["v0.3.1"]
"sha256:7a4d062fed0dd80d216f94803e7c6bfb06022ae76aa0960602878d964fa9fdd2": ["v0.4.0"]
"sha256:d91229530a784c0569adf7192978f64c9371e906ed726cc3061aa98c2706bdce": ["v0.5.0"]
"sha256:1f004a487c09c76ebf14d1dc01434aa17e31e39def56451ec5727160fdcd0017": ["v0.6.0"]
- name: octodns
dmap:
"sha256:838d6134142eea4ce926c0bfe2f2bd2bbd7580e42e42aea39259575951273c44": ["v20200501-36789b1"]
Expand Down