Skip to content

Add autoscaling and set higher cpu for raster-eoapi service#42

Merged
spwoodcock merged 3 commits intomainfrom
raster-autoscaling
Mar 21, 2026
Merged

Add autoscaling and set higher cpu for raster-eoapi service#42
spwoodcock merged 3 commits intomainfrom
raster-autoscaling

Conversation

@dakotabenjamin
Copy link
Copy Markdown
Member

What type of PR is this? (check all applicable)

  • 🍕 Feature
  • 🐛 Bug Fix
  • 📝 Documentation
  • 🧑‍💻 Refactor
  • ✅ Test
  • 🤖 Build or CI
  • ❓ Other (please specify)

Describe this PR

The raster renderer has been crashing with a single user (me) attempting to use calls such as

[https://api.imagery.hotosm.org/raster/searches/867568a0734dd1f665da93a5a5e2a5ec/tiles/WebMercatorQuad/{z}/{x}/{y}@1x.png?assets=visual](https://api.imagery.hotosm.org/raster/searches/867568a0734dd1f665da93a5a5e2a5ec/tiles/WebMercatorQuad/%7Bz%7D/%7Bx%7D/%7By%7D@1x.png?assets=visual)

Default cpu/mem config for the pods:

     Limits:
      cpu:     768m
      memory:  4Gi
    Requests:
      cpu:      256m
      memory:   3Gi

0.25 of a vCPU is probably way too small, new config (below) should fix most of the problems.

      requests:
        cpu: "1024m"
        memory: "3Gi"
      limits:
        cpu: "2048m"
        memory: "4Gi"

Additionally, we enable autoscaling for the raster-eoapi pod. Current scaling target is 85% CPU usage- This needs discussion, and potentially load-testing.

Screenshots

grafana dashboard showing some of the metrics leading to these changes:
image

@github-actions
Copy link
Copy Markdown

github-actions bot commented Nov 5, 2025

tofu apply -chdir=terraform tfplan
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
By @spwoodcock at 2026-03-21T04:29:45Z (view log).
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

cluster_name = "hotosm-production-cluster"
karpenter_controller_role_arn = "arn:aws:iam::670261699094:role/hotosm-production-karpenter-controller"
karpenter_interruption_queue_name = "hotosm-production-cluster"
karpenter_node_instance_profile_name = "KarpenterNodeRole-hotosm-production-cluster"
s3_backup_role = "arn:aws:iam::670261699094:role/hotosm-production-bucket-access"

Comment thread kubernetes/helm/eoapi-values.yaml
Comment thread kubernetes/helm/eoapi-values.yaml Outdated
@spwoodcock spwoodcock force-pushed the main branch 6 times, most recently from 3bc2c46 to ea89cf6 Compare February 11, 2026 17:32
@spwoodcock spwoodcock force-pushed the main branch 2 times, most recently from cb903f7 to f5e125e Compare March 21, 2026 03:07
@spwoodcock spwoodcock marked this pull request as ready for review March 21, 2026 04:28
@spwoodcock spwoodcock merged commit b8037d0 into main Mar 21, 2026
1 check passed
@spwoodcock spwoodcock deleted the raster-autoscaling branch March 21, 2026 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants