Skip to content

Different rate limits depending on HTTP method (#5555) #5869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
achave11-ucsc merged 17 commits into from
Jul 15, 2025
Merged

Different rate limits depending on HTTP method (#5555) #5869

achave11-ucsc merged 17 commits into from
Jul 15, 2025

Conversation

@dsotirho-ucsc
Copy link
Contributor

@dsotirho-ucsc dsotirho-ucsc commented Jan 19, 2024 •
edited by achave11-ucsc
Loading

Connected issues: #5555

Checklist

Author

  • PR is a draft
  • Target branch is develop
  • Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
  • On ZenHub, PR is connected to all issues it (partially) resolves
  • PR description links to connected issues
  • PR title matches1 that of a connected issue or comment in PR explains why they're different
  • PR title references all connected issues
  • For each connected issue, there is at least one commit whose title references that issue

1 when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (partiality)

  • Added p tag to titles of partial commits
  • This PR is labeled partial or completely resolves all connected issues
  • This PR partially resolves each of the connected issues or does not have the partial label

Author (chains)

  • This PR is blocked by previous PR in the chain or is not chained to another PR
  • The blocking PR is labeled base or this PR is not chained to another PR
  • This PR is labeled chained or is not chained to another PR

Author (reindex, API changes)

  • Added r tag to commit title or the changes introduced by this PR will not require reindexing of any deployment
  • This PR is labeled reindex:dev or the changes introduced by it will not require reindexing of dev
  • This PR is labeled reindex:anvildev or the changes introduced by it will not require reindexing of anvildev
  • This PR is labeled reindex:anvilprod or the changes introduced by it will not require reindexing of anvilprod
  • This PR is labeled reindex:prod or the changes introduced by it will not require reindexing of prod
  • This PR is labeled reindex:partial and its description documents the specific reindexing procedure for dev, anvildev, anvilprod and prod or requires a full reindex or carries none of the labels reindex:dev, reindex:anvildev, reindex:anvilprod and reindex:prod
  • This PR and its connected issues are labeled API or this PR does not modify a REST API
  • Added a (A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST API
  • Updated REST API version number in app.py or this PR does not modify a REST API

Author (upgrading deployments)

  • Ran make image_manifests.json and committed the resulting changes or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable
  • Documented upgrading of deployments in UPGRADING.rst or this PR does not require upgrading deployments
  • Added u tag to commit title or this PR does not require upgrading deployments
  • This PR is labeled upgrade or does not require upgrading deployments
  • This PR is labeled deploy:shared or does not modify image_manifests.json, and does not require deploying the shared component for any other reason
  • This PR is labeled deploy:gitlab or does not require deploying the gitlab component
  • This PR is labeled deploy:runner or does not require deploying the runner image

Author (hotfixes)

  • Added F tag to main commit title or this PR does not include permanent fix for a temporary hotfix
  • Reverted the temporary hotfixes for any connected issues or the none of the stable branches (anvilprod and prod) have temporary hotfixes for any of the issues connected to this PR

Author (before every review)

  • Rebased PR branch on develop, squashed old fixups
  • Ran make requirements_update or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile
  • Added R tag to commit title or this PR does not modify requirements*.txt
  • This PR is labeled reqs or does not modify requirements*.txt
  • make integration_test passes in personal deployment or this PR does not modify functionality that could affect the IT outcome

Peer reviewer (after approval)

  • PR is not a draft
  • Ticket is in Review requested column
  • PR is awaiting requested review from system administrator
  • PR is assigned to only the system administrator

System administrator (after approval)

  • Actually approved the PR
  • Labeled connected issues as demo or no demo
  • Commented on connected issues about demo expectations or all connected issues are labeled no demo
  • Decided if PR can be labeled no sandbox
  • A comment to this PR details the completed security design review
  • PR title is appropriate as title of merge commit
  • N reviews label is accurate
  • Moved ticket to Approved column
  • PR is assigned to only the operator

Operator (before pushing merge the commit)

  • Checked reindex:… labels and r commit title tag
  • Checked that demo expectations are clear or all connected issues are labeled no demo
  • Squashed PR branch and rebased onto develop
  • Sanity-checked history
  • Pushed PR branch to GitHub
  • Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused or this PR is not labeled deploy:shared
  • Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply or this PR is not labeled deploy:gitlab
  • Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused or this PR is not labeled deploy:shared
  • Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply or this PR is not labeled deploy:gitlab
  • Checked the items in the next section or this PR is labeled deploy:gitlab
  • PR is assigned to only the system administrator or this PR is not labeled deploy:gitlab

System administrator

  • Background migrations for dev.gitlab are complete or this PR is not labeled deploy:gitlab
  • Background migrations for anvildev.gitlab are complete or this PR is not labeled deploy:gitlab
  • PR is assigned to only the operator

Operator (before pushing merge the commit)

  • Ran _select dev.gitlab && make -C terraform/gitlab/runner or this PR is not labeled deploy:runner
  • Ran _select anvildev.gitlab && make -C terraform/gitlab/runner or this PR is not labeled deploy:runner
  • Added sandbox label or PR is labeled no sandbox
  • Pushed PR branch to GitLab dev or PR is labeled no sandbox
  • Pushed PR branch to GitLab anvildev or PR is labeled no sandbox
  • Build passes in sandbox deployment or PR is labeled no sandbox
  • Build passes in anvilbox deployment or PR is labeled no sandbox
  • Reviewed build logs for anomalies in sandbox deployment or PR is labeled no sandbox
  • Reviewed build logs for anomalies in anvilbox deployment or PR is labeled no sandbox
  • Deleted unreferenced indices in sandbox or this PR does not remove catalogs or otherwise causes unreferenced indices in dev
  • Deleted unreferenced indices in anvilbox or this PR does not remove catalogs or otherwise causes unreferenced indices in anvildev
  • Started reindex in sandbox or this PR is not labeled reindex:dev
  • Started reindex in anvilbox or this PR is not labeled reindex:anvildev
  • Checked for failures in sandbox or this PR is not labeled reindex:dev
  • Checked for failures in anvilbox or this PR is not labeled reindex:anvildev
  • The title of the merge commit starts with the title of this PR
  • Added PR # reference to merge commit title
  • Collected commit title tags in merge commit title but only included p if the PR is also labeled partial
  • Moved connected issues to Merged column in ZenHub
  • Pushed merge commit to GitHub

Operator (chain shortening)

  • Changed the target branch of the blocked PR to develop or this PR is not labeled base
  • Removed the chained label from the blocked PR or this PR is not labeled base
  • Removed the blocking relationship from the blocked PR or this PR is not labeled base
  • Removed the base label from this PR or this PR is not labeled base

Operator (after pushing the merge commit)

  • Pushed merge commit to GitLab dev
  • Pushed merge commit to GitLab anvildev
  • Build passes on GitLab dev
  • Reviewed build logs for anomalies on GitLab dev
  • Build passes on GitLab anvildev
  • Reviewed build logs for anomalies on GitLab anvildev
  • Ran _select dev.shared && make -C terraform/shared apply or this PR is not labeled deploy:shared
  • Ran _select anvildev.shared && make -C terraform/shared apply or this PR is not labeled deploy:shared
  • Deleted PR branch from GitHub
  • Deleted PR branch from GitLab dev
  • Deleted PR branch from GitLab anvildev

Operator (reindex)

  • Deindexed all unreferenced catalogs in dev or this PR is neither labeled reindex:partial nor reindex:dev
  • Deindexed all unreferenced catalogs in anvildev or this PR is neither labeled reindex:partial nor reindex:anvildev
  • Deindexed specific sources in dev or this PR is neither labeled reindex:partial nor reindex:dev
  • Deindexed specific sources in anvildev or this PR is neither labeled reindex:partial nor reindex:anvildev
  • Indexed specific sources in dev or this PR is neither labeled reindex:partial nor reindex:dev
  • Indexed specific sources in anvildev or this PR is neither labeled reindex:partial nor reindex:anvildev
  • Started reindex in dev or this PR does not require reindexing dev
  • Started reindex in anvildev or this PR does not require reindexing anvildev
  • Checked for, triaged and possibly requeued messages in both fail queues in dev or this PR does not require reindexing dev
  • Checked for, triaged and possibly requeued messages in both fail queues in anvildev or this PR does not require reindexing anvildev
  • Emptied fail queues in dev or this PR does not require reindexing dev
  • Emptied fail queues in anvildev or this PR does not require reindexing anvildev

Operator

  • Propagated the deploy:shared, deploy:gitlab, deploy:runner, reindex:partial, reindex:anvilprod and reindex:prod labels to the next promotion PRs or this PR carries none of these labels
  • Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner, reindex:partial, reindex:anvilprod and reindex:prod labels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labels
  • PR is assigned to no one

Shorthand for review comments

  • L line is too long
  • W line wrapping is wrong
  • Q bad quotes
  • F other formatting problem

@coveralls
Copy link

coveralls commented Jan 19, 2024 •
edited
Loading

Coverage Status

coverage: 85.217% (-0.02%) from 85.239%
when pulling f16a7ea on issues/dsotirho-ucsc/5555-rate-limit-on-http-method
into e4652f5 on develop.

@codecov
Copy link

codecov bot commented Jan 19, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 66.66667% with 20 lines in your changes missing coverage. Please review.

Project coverage is 85.03%. Comparing base (e4652f5) to head (f16a7ea).
Report is 18 commits behind head on develop.

Files with missing lines Patch % Lines
src/azul/template/__main__.py 0.00% 9 Missing ⚠️
src/azul/deployment.py 46.15% 7 Missing ⚠️
test/integration_test.py 0.00% 3 Missing ⚠️
src/azul/queues.py 0.00% 1 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff             @@
##           develop    #5869      +/-   ##
===========================================
- Coverage    85.05%   85.03%   -0.03%     
===========================================
  Files          155      156       +1     
  Lines        22447    22460      +13     
===========================================
+ Hits         19093    19099       +6     
- Misses        3354     3361       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 2 times, most recently from 34336cf to 1259465 Compare January 19, 2024 19:09
@dsotirho-ucsc
Copy link
Contributor Author

dsotirho-ucsc commented Jan 19, 2024

Tested against personal deployment

Using watch to flood API with requests:

daniel@Crispin ~ $ curl -X 'POST' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/index/summary?catalog=dcp3'
{"message":"Missing Authentication Token"}daniel@Crispin ~ $
daniel@Crispin ~ $
daniel@Crispin ~ $ watch -n 1 curl -X 'POST' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/index/summary?catalog=dcp3'
daniel@Crispin ~ $

Showing POST requests are blocked while GET request are allowed:

daniel@Crispin ~ $ curl -X 'POST' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/index/summary?catalog=dcp3'
{"message":"Forbidden"}daniel@Crispin ~ $
daniel@Crispin ~ $
daniel@Crispin ~ $ curl 'https://service.daniel.dev.singlecell.gi.ucsc.edu/index/summary?catalog=dcp3'
{"projectCount":105,"specimenCount":889,"speciesCount":2,"fileCount":3440,"totalFileSize":10967869203586.0,"donorCount":653,"labCount":204,"organTypes":["blood","brain","skin of body","kidney","large intestine","gonad","pancreas","hematopoietic system","central nervous system","endometrium","heart","lung","embryo","eye","lymph node","placenta","ovary","decidua","umbilical cord","colon","digestive system","liver","thymus","retina","tumor",null,"adipose tissue","arterial blood vessel","testis","tonsil","hindlimb","oral cavity","whole embryos","blastocyst","immune system","muscle","nose","skeletal muscle organ","skin","spleen","stem cell","uterus","wall of lateral ventricle","abdomen","immune organ","mouse kidney","small intestine","tail","trachea"],"fileTypeSummaries":[{"count":2885,"totalSize":7838690358439.0,"matrixCellCount":0.0,"format":"fastq.gz"},{"count":154,"totalSize":197202319290.0,"matrixCellCount":0.0,"format":"loom"},{"count":129,"totalSize":2395989149587.0,"matrixCellCount":0.0,"format":"bam"},{"count":83,"totalSize":506850977636.0,"matrixCellCount":0.0,"format":"fastq"},{"count":32,"totalSize":2298010985.0,"matrixCellCount":0.0,"format":"txt.gz"},{"count":29,"totalSize":156508385.0,"matrixCellCount":0.0,"format":"tsv.gz"},{"count":24,"totalSize":4228874240.0,"matrixCellCount":0.0,"format":"tar"},{"count":15,"totalSize":13529137.0,"matrixCellCount":160631.0,"format":"txt"},{"count":14,"totalSize":265348615.0,"matrixCellCount":51516.0,"format":"tsv"},{"count":12,"totalSize":38143884.0,"matrixCellCount":0.0,"format":"pdf"},{"count":11,"totalSize":8367867567.0,"matrixCellCount":0.0,"format":"zip"},{"count":10,"totalSize":603217402.0,"matrixCellCount":0.0,"format":"csv"},{"count":9,"totalSize":17863772.0,"matrixCellCount":0.0,"format":"csv.gz"},{"count":8,"totalSize":1921436840.0,"matrixCellCount":0.0,"format":"mtx.gz"},{"count":6,"totalSize":1739858713.0,"matrixCellCount":51516.0,"format":"mtx"},{"count":5,"totalSize":6764456.0,"matrixCellCount":0.0,"format":"xlsx"},{"count":4,"totalSize":3960869846.0,"matrixCellCount":800000.0,"format":"h5ad"},{"count":3,"totalSize":2637887113.0,"matrixCellCount":162420.0,"format":"Rds"},{"count":2,"totalSize":8961366.0,"matrixCellCount":0.0,"format":"h5"},{"count":2,"totalSize":1686428099.0,"matrixCellCount":0.0,"format":"rds"},{"count":1,"totalSize":719215660.0,"matrixCellCount":0.0,"format":"Robj"},{"count":1,"totalSize":428090368.0,"matrixCellCount":0.0,"format":"rds.gz"},{"count":1,"totalSize":37522186.0,"matrixCellCount":0.0,"format":"tar.gz"}],"cellCountSummaries":[{"organType":["hematopoietic system"],"countOfDocsWithOrganType":9637,"totalCellCountByOrgan":1489527.0},{"organType":["endometrium"],"countOfDocsWithOrganType":2157,"totalCellCountByOrgan":2148.0},{"organType":["blood"],"countOfDocsWithOrganType":208,"totalCellCountByOrgan":913740.0},{"organType":["skin of body"],"countOfDocsWithOrganType":104,"totalCellCountByOrgan":15460.0},{"organType":["brain"],"countOfDocsWithOrganType":93,"totalCellCountByOrgan":558844.0},{"organType":["pancreas"],"countOfDocsWithOrganType":77,"totalCellCountByOrgan":52358.0},{"organType":["kidney"],"countOfDocsWithOrganType":71,"totalCellCountByOrgan":163820.0},{"organType":["large intestine"],"countOfDocsWithOrganType":67,"totalCellCountByOrgan":6.0},{"organType":["gonad"],"countOfDocsWithOrganType":43,"totalCellCountByOrgan":43.0},{"organType":["umbilical cord"],"countOfDocsWithOrganType":42,"totalCellCountByOrgan":20032.0},{"organType":["central nervous system"],"countOfDocsWithOrganType":30,"totalCellCountByOrgan":34400.0},{"organType":["decidua"],"countOfDocsWithOrganType":28,"totalCellCountByOrgan":14000.0},{"organType":["heart"],"countOfDocsWithOrganType":28,"totalCellCountByOrgan":90000.0},{"organType":["lung"],"countOfDocsWithOrganType":25,"totalCellCountByOrgan":21775.0},{"organType":["colon"],"countOfDocsWithOrganType":21,"totalCellCountByOrgan":4744.0},{"organType":["embryo"],"countOfDocsWithOrganType":18,"totalCellCountByOrgan":15.0},{"organType":["eye"],"countOfDocsWithOrganType":16,"totalCellCountByOrgan":53535.0},{"organType":["ovary"],"countOfDocsWithOrganType":16,"totalCellCountByOrgan":11.0},{"organType":["lymph node"],"countOfDocsWithOrganType":14,"totalCellCountByOrgan":8.0},{"organType":["placenta"],"countOfDocsWithOrganType":14,"totalCellCountByOrgan":22526.0},{"organType":["testis"],"countOfDocsWithOrganType":14,"totalCellCountByOrgan":7511.0},{"organType":["abdomen"],"countOfDocsWithOrganType":13,"totalCellCountByOrgan":13.0},{"organType":["liver"],"countOfDocsWithOrganType":11,"totalCellCountByOrgan":30006.0},{"organType":["arterial blood vessel"],"countOfDocsWithOrganType":10,"totalCellCountByOrgan":0.0},{"organType":["digestive system"],"countOfDocsWithOrganType":9,"totalCellCountByOrgan":72000.0},{"organType":["thymus"],"countOfDocsWithOrganType":9,"totalCellCountByOrgan":0.0},{"organType":[null],"countOfDocsWithOrganType":9,"totalCellCountByOrgan":30000.0},{"organType":["tumor"],"countOfDocsWithOrganType":8,"totalCellCountByOrgan":8.0},{"organType":["retina"],"countOfDocsWithOrganType":6,"totalCellCountByOrgan":8217.0},{"organType":["adipose tissue"],"countOfDocsWithOrganType":5,"totalCellCountByOrgan":0.0},{"organType":["oral cavity"],"countOfDocsWithOrganType":4,"totalCellCountByOrgan":4.0},{"organType":["stem cell"],"countOfDocsWithOrganType":4,"totalCellCountByOrgan":0.0},{"organType":["tonsil"],"countOfDocsWithOrganType":4,"totalCellCountByOrgan":0.0},{"organType":["trachea"],"countOfDocsWithOrganType":4,"totalCellCountByOrgan":4.0},{"organType":["hindlimb"],"countOfDocsWithOrganType":3,"totalCellCountByOrgan":11590.0},{"organType":["skin"],"countOfDocsWithOrganType":3,"totalCellCountByOrgan":2413.0},{"organType":["whole embryos"],"countOfDocsWithOrganType":3,"totalCellCountByOrgan":0.0},{"organType":["blastocyst"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":2.0},{"organType":["immune system"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["muscle"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":3214.0},{"organType":["nose"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["skeletal muscle organ"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["spleen"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":30043.0},{"organType":["uterus"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["wall of lateral ventricle"],"countOfDocsWithOrganType":2,"totalCellCountByOrgan":0.0},{"organType":["immune organ"],"countOfDocsWithOrganType":1,"totalCellCountByOrgan":1.0},{"organType":["mouse kidney"],"countOfDocsWithOrganType":1,"totalCellCountByOrgan":5648.0},{"organType":["small intestine"],"countOfDocsWithOrganType":1,"totalCellCountByOrgan":0.0},{"organType":["tail"],"countOfDocsWithOrganType":1,"totalCellCountByOrgan":1.0}],"projects":[{"projects":{"estimatedCellCount":12086.0},"cellSuspensions":{"totalCells":null}},{"projects":{"estimatedCellCount":null},"cellSuspensions":{"totalCells":2559060.0}},{"projects":{"estimatedCellCount":927877.0},"cellSuspensions":{"totalCells":1093788.0}}]}daniel@Crispin ~ $

Alarm raised:
Screenshot 2024-01-19 at 11 20 41 AM

@hannes-ucsc
Copy link
Member

hannes-ucsc commented Jan 19, 2024

watch is inadequate for this experiment because it it sequential, not parallel.

@hannes-ucsc
Copy link
Member

hannes-ucsc commented Jan 19, 2024

Also, /index/summary does not support POST, only GET, so the 403 is not from the WAF, but from API Gateway. So I'm not sure what you are testing for here. We discussed 5, 9, 10, 11 and 15 concurrent requests against the manifest endpoint with an inconsequential variation in the filters parameter so that you get different manifest keys and thereby subvert the caching algorithm. That's not at all what you are doing here.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 1259465 to caad149 Compare January 20, 2024 00:41
@dsotirho-ucsc
Copy link
Contributor Author

dsotirho-ucsc commented Jan 22, 2024

Also, /index/summary does not support POST, only GET, so the 403 is not from the WAF, but from API Gateway. So I'm not sure what you are testing for here. We discussed 5, 9, 10, 11 and 15 concurrent requests against the manifest endpoint with an inconsequential variation in the filters parameter so that you get different manifest keys and thereby subvert the caching algorithm. That's not at all what you are doing here.

The above test was run prior to deciding to run a test with concurrent requests.

While /index/summary doesn't support POST, it sufficed for the test since the WAF can still block these requests. Note the response from the initial request is

{"message":"Missing Authentication Token"}

and after the WAF starts blocking the requests, the response is

{"message":"Forbidden"}

Also, it's actually PUT, not POST we want to block as an expensive request. This will be fixed before the next test with concurrent requests.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 2 times, most recently from 2a95628 to 0139402 Compare January 22, 2024 23:18
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 0139402 to 13e373d Compare January 23, 2024 00:27
@dsotirho-ucsc
Copy link
Contributor Author

dsotirho-ucsc commented Jan 23, 2024

Test results making concurrent manifest generation requests with the manifest Lambda's Reserved concurrency set to 10.

5 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 1 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 4 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 1 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 3 got a 301, retrying after 4 seconds
Thread 2 got a 301, retrying after 6 seconds
Thread 1 got a 302, done.
Thread 4 got a 301, retrying after 6 seconds
Thread 5 got a 302, done.
Thread 3 got a 301, retrying after 6 seconds
Thread 2 got a 302, done.
Thread 4 got a 302, done.
Thread 3 got a 302, done.
9 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 6 making initial PUT request
Thread 7 making initial PUT request
Thread 8 making initial PUT request
Thread 9 making initial PUT request
Thread 9 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 9 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 3 got a 301, retrying after 4 seconds
Thread 1 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 4 seconds
Thread 6 got a 301, retrying after 4 seconds
Thread 8 got a 301, retrying after 4 seconds
Thread 9 got a 302, done.
Thread 4 got a 302, done.
Thread 2 got a 302, done.
Thread 3 got a 302, done.
Thread 1 got a 302, done.
Thread 7 got a 302, done.
Thread 5 got a 302, done.
Thread 6 got a 301, retrying after 6 seconds
Thread 8 got a 301, retrying after 6 seconds
Thread 6 got a 302, done.
Thread 8 got a 302, done.
10 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 6 making initial PUT request
Thread 7 making initial PUT request
Thread 8 making initial PUT request
Thread 9 making initial PUT request
Thread 10 making initial PUT request
Thread 2 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 8 got a 301, retrying after 4 seconds
Thread 6 got a 301, retrying after 4 seconds
Thread 10 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 9 got a 301, retrying after 4 seconds
Thread 3 got a 301, retrying after 4 seconds
Thread 1 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 4 seconds
Thread 2 got a 302, done.
Thread 8 got a 302, done.
Thread 10 got a 302, done.
Thread 4 got a 302, done.
Thread 6 got a 302, done.
Thread 3 got a 302, done.
Thread 9 got a 302, done.
Thread 1 got a 302, done.
Thread 7 got a 302, done.
Thread 5 got a 302, done.
11 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 6 making initial PUT request
Thread 7 making initial PUT request
Thread 8 making initial PUT request
Thread 9 making initial PUT request
Thread 10 making initial PUT request
Thread 11 making initial PUT request
Thread 9 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 3 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 5 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 5 got a 301, retrying after 4 seconds
Thread 11 got a 301, retrying after 4 seconds
Thread 10 got a 301, retrying after 4 seconds
Thread 7 got a 301, retrying after 4 seconds
Thread 9 got a 301, retrying after 4 seconds
Thread 6 got a 301, retrying after 4 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 8 got a 301, retrying after 4 seconds
Thread 4 got a 302, done.
Thread 1 got a 302, done.
Thread 7 got a 302, done.
Thread 11 got a 302, done.
Thread 5 got a 302, done.
Thread 10 got a 302, done.
Thread 2 got a 302, done.
Thread 6 got a 302, done.
Thread 8 got a 302, done.
Thread 9 got a 302, done.
15 concurrent threads 
Thread 1 making initial PUT request
Thread 2 making initial PUT request
Thread 3 making initial PUT request
Thread 4 making initial PUT request
Thread 5 making initial PUT request
Thread 6 making initial PUT request
Thread 7 making initial PUT request
Thread 8 making initial PUT request
Thread 9 making initial PUT request
Thread 10 making initial PUT request
Thread 11 making initial PUT request
Thread 12 making initial PUT request
Thread 13 making initial PUT request
Thread 14 making initial PUT request
Thread 15 making initial PUT request
Thread 4 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 15 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 14 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 12 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 1 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 14 got a 301, retrying after 1 seconds
Thread 7 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 15 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 12 got a 301, retrying after 1 seconds
Thread 1 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 7 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 1 seconds
Thread 14 got a 301, retrying after 1 seconds
Thread 4 got a 301, retrying after 1 seconds
Thread 3 got a 301, retrying after 1 seconds
Thread 15 got a 301, retrying after 1 seconds
Thread 2 got a 301, retrying after 1 seconds
Thread 11 got a 301, retrying after 1 seconds
Thread 12 got a 301, retrying after 1 seconds
Thread 9 got a 301, retrying after 1 seconds
Thread 5 got a 301, retrying after 1 seconds
Thread 8 got a 301, retrying after 1 seconds
Thread 10 got a 301, retrying after 1 seconds
Thread 6 got a 301, retrying after 4 seconds
Thread 7 got a 301, retrying after 4 seconds
Thread 14 got a 301, retrying after 4 seconds
Thread 4 got a 301, retrying after 4 seconds
Thread 13 got a 301, retrying after 1 seconds
Thread 12 got a 301, retrying after 4 seconds
Thread 11 got a 301, retrying after 4 seconds
Thread 3 got a 301, retrying after 4 seconds
Thread 2 got a 301, retrying after 4 seconds
Thread 15 got a 301, retrying after 4 seconds
Thread 9 got a 301, retrying after 4 seconds
Thread 5 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 8 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 10 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 13 got a 500 {"Code":"ChaliceViewError","Message":"('Failed to generate manifest', 'FAILED', None)"}
Thread 4 got a 302, done.
Thread 6 got a 302, done.
Thread 14 got a 302, done.
Thread 3 got a 302, done.
Thread 7 got a 302, done.
Thread 12 got a 302, done.
Thread 11 got a 302, done.
Thread 9 got a 302, done.
Thread 2 got a 302, done.
Thread 15 got a 302, done.

@hannes-ucsc
Copy link
Member

hannes-ucsc commented Jan 23, 2024

It's impossible to reproduce your experiment without the source to the script that you presumably used. Please repeat the experiment with a simple command line that uses rush or gargs for running commands in parallel and curl for making the actual request. Prepare one file with 15 manifest URLs, one per line, and use head to limit the number of URLs that gets passed to gargs or rush. Post the file contents and the command line that was used. For one of the failed requests, please post relevant log entries from the service and manifest lambdas.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 6 times, most recently from 4c4bfc6 to 9b79b0a Compare January 30, 2024 17:33
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch 3 times, most recently from 2d5008a to 3366611 Compare February 3, 2024 00:02
@dsotirho-ucsc
Copy link
Contributor Author

dsotirho-ucsc commented Feb 3, 2024

It's impossible to reproduce your experiment without the source to the script that you presumably used. Please repeat the experiment with a simple command line that uses rush or gargs for running commands in parallel and curl for making the actual request. Prepare one file with 15 manifest URLs, one per line, and use head to limit the number of URLs that gets passed to gargs or rush. Post the file contents and the command line that was used. For one of the failed requests, please post relevant log entries from the service and manifest lambdas.

Tested 15 concurrent manifest requests with the manifest Lambda concurrency set to 10.

manifest_urls.txt

https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10001000000%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000001%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000002%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000003%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000004%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000005%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000006%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000007%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000008%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000009%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000010%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000011%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000012%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000013%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000014%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000015%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000016%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000017%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000018%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000019%5D%5D%7D%7D&format=curl
https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000020%5D%5D%7D%7D&format=curl

Command line

head -n 15 manifest_urls.txt | xargs -P 15 -I % -S 512 python -c "from furl import furl; from integration_test import IndexingIntegrationTest as iit; from azul.logging import configure_script_logging; configure_script_logging(); t = iit(); t.setUp(); m = t._get_url_content('PUT', furl('%'))"

Terminal log

(abridged to show the all initial requests, and then the thread for 1 failed result)
(full terminal log: 5555-test-terminal-log.txt)

(.venv) daniel@Crispin ~/repo/azul1 $ head -n 15 manifest_urls.txt | xargs -P 15 -I % -S 512 python -c "from furl import furl; from integration_test import IndexingIntegrationTest as iit; from azul.logging import configure_script_logging; configure_script_logging(); t = iit(); t.setUp(); m = t._get_url_content('PUT', furl('%'))"
2024-02-02 11:17:37,682    INFO MainThread test.integration_test: Using random seed 9200391225568754709
2024-02-02 11:17:37,682    INFO MainThread test.integration_test: Using random seed 4956485098573584590
2024-02-02 11:17:37,682    INFO MainThread test.integration_test: Using random seed 8727454292430743500
2024-02-02 11:17:37,683    INFO MainThread test.integration_test: Using random seed 343936599250221468
2024-02-02 11:17:37,683    INFO MainThread test.integration_test: Using random seed 714364583535640742
2024-02-02 11:17:37,684    INFO MainThread test.integration_test: Using random seed 3746667822296243315
2024-02-02 11:17:37,683    INFO MainThread test.integration_test: Using random seed 7430512986588854954
2024-02-02 11:17:37,685    INFO MainThread test.integration_test: Using random seed 3706013619302031698
2024-02-02 11:17:37,685    INFO MainThread test.integration_test: Using random seed 8406687919479253520
2024-02-02 11:17:37,686    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000009%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,686    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000003%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,687    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000002%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,688    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000004%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,688    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000013%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,686    INFO MainThread test.integration_test: Using random seed 5971879927632929552
2024-02-02 11:17:37,691    INFO MainThread test.integration_test: Using random seed 2449699439315315317
2024-02-02 11:17:37,691    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000005%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,691    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000001%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,691    INFO MainThread test.integration_test: Using random seed 4220658395087540698
2024-02-02 11:17:37,693    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000008%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,684    INFO MainThread test.integration_test: Using random seed 2359823532252616098
2024-02-02 11:17:37,694    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000014%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,695    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000011%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,696    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000010%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,697    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10001000000%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,698    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000007%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,698    INFO MainThread test.integration_test: Using random seed 812345280441374550
2024-02-02 11:17:37,700    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000006%5D%5D%7D%7D&format=curl'
2024-02-02 11:17:37,702    INFO MainThread test.integration_test: Using random seed 8019449667869935403
2024-02-02 11:17:37,705    INFO MainThread test.integration_test: Making PUT request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000012%5D%5D%7D%7D&format=curl'
…
2024-02-02 11:17:42,734    INFO MainThread test.integration_test: Got 301 response after 5.046s from PUT to https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C10002000013%5D%5D%7D%7D&format=curl
2024-02-02 11:17:42,734    INFO MainThread test.integration_test: Sleeping 1.000s to honor Retry-After header
…
2024-02-02 11:17:43,736    INFO MainThread test.integration_test: Making GET request to 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ=='
…
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 795, in _get_url_content
    response = self._get_url(method, url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 812, in _get_url
    response = self._get_url_unchecked(method, url, stream=stream)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/test/integration_test.py", line 840, in _get_url_unchecked
    response = http.request(method=method,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/request.py", line 77, in request
    return self.request_encode_url(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/request.py", line 99, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/src/azul/http.py", line 68, in urlopen
    response = super().urlopen(method, url, body=body, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/src/azul/http.py", line 42, in urlopen
    return self._inner.urlopen(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 894, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 894, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 894, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 884, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='service.daniel.dev.singlecell.gi.ucsc.edu', port=443): Max retries exceeded with url: /manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ== (Caused by ResponseError('too many 500 error responses'))
…

CloudWatch Log Insights (/aws/lambda/azul-service-daniel)

| filter @message like /START|END|Received|Returning/

[
    {
        "@timestamp": "2024-02-02 19:17:42.156",
        "@message": "START RequestId: bbbe077b-6e8e-4b66-b6b9-d0afdb9d7eac Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:42.157",
        "@message": "[INFO]\t2024-02-02T19:17:42.157Z\tbbbe077b-6e8e-4b66-b6b9-d0afdb9d7eac\tazul.chalice\tReceived PUT request for '/manifest/files', with {\"query\": {\"catalog\": \"dcp3\", \"filters\": \"{\\\"fileSize\\\":{\\\"within\\\":[[1,10002000013]]}}\", \"format\": \"curl\"}, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"gbKNKEkubbCV8tXHfDBD3_d0V9wE6Rzq_ot1uQ5NpNY9esh5Fd-cRg==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd2-01f04c33326720b013c99034\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.154\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:42.766",
        "@message": "[DEBUG]\t2024-02-02T19:17:42.765Z\tbbbe077b-6e8e-4b66-b6b9-d0afdb9d7eac\tazul.chalice\tReturning 301 response with headers {\"Location\": \"https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==\", \"Retry-After\": \"1\", \"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\nnull"
    },
    {
        "@timestamp": "2024-02-02 19:17:42.769",
        "@message": "END RequestId: bbbe077b-6e8e-4b66-b6b9-d0afdb9d7eac\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:43.876",
        "@message": "START RequestId: a4c11e0f-7273-4047-9b0e-e48a9ac28780 Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:43.876",
        "@message": "[INFO]\t2024-02-02T19:17:43.876Z\ta4c11e0f-7273-4047-9b0e-e48a9ac28780\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"WKeYDnUxwOuawSOOCe-Lg3HuXBZ3D13DAxOD54DUDYa0YSTMg-Bz3Q==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd7-7f6601c96a399fa102e4fce0\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.106\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:43.911",
        "@message": "[DEBUG]\t2024-02-02T19:17:43.911Z\ta4c11e0f-7273-4047-9b0e-e48a9ac28780\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:43.913",
        "@message": "END RequestId: a4c11e0f-7273-4047-9b0e-e48a9ac28780\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.145",
        "@message": "START RequestId: 8da73087-4dd3-4018-ab8e-5f10a578c5f0 Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.146",
        "@message": "[INFO]\t2024-02-02T19:17:44.146Z\t8da73087-4dd3-4018-ab8e-5f10a578c5f0\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"ViQlGZVfgAtGOEN6OZDK5laHXKryLwsr5O8ny6X-h7mgLtV1VZmG7w==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-396f8557067115166b3e2dc7\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.84\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.177",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.177Z\t8da73087-4dd3-4018-ab8e-5f10a578c5f0\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.179",
        "@message": "END RequestId: 8da73087-4dd3-4018-ab8e-5f10a578c5f0\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.378",
        "@message": "START RequestId: 75fae844-be68-4a5e-ae0f-8dfb9a6e32d9 Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.378",
        "@message": "[INFO]\t2024-02-02T19:17:44.378Z\t75fae844-be68-4a5e-ae0f-8dfb9a6e32d9\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"aYs4D_kDPVQCh0k_EatqK5N9vttmbzS9dKaUrODgxzo3hDXNrOqIsA==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-1514d5b642dc81e70fc6a038\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.154\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.413",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.412Z\t75fae844-be68-4a5e-ae0f-8dfb9a6e32d9\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.414",
        "@message": "END RequestId: 75fae844-be68-4a5e-ae0f-8dfb9a6e32d9\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.547",
        "@message": "START RequestId: c0d907be-9972-49c5-8a8e-8173c811e8ff Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.548",
        "@message": "[INFO]\t2024-02-02T19:17:44.548Z\tc0d907be-9972-49c5-8a8e-8173c811e8ff\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"mGFP3_i2TW9UjYBkeCnMCuUZWUm7Db-y2JfG_lKCUvbq39iOUjE93g==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-73b89351210a323c48ebbc73\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.161\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.578",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.578Z\tc0d907be-9972-49c5-8a8e-8173c811e8ff\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.579",
        "@message": "END RequestId: c0d907be-9972-49c5-8a8e-8173c811e8ff\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.678",
        "@message": "START RequestId: 21856a47-aa92-46d7-827e-24b4cdfb0e3b Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.679",
        "@message": "[INFO]\t2024-02-02T19:17:44.679Z\t21856a47-aa92-46d7-827e-24b4cdfb0e3b\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"tmIyiwQANcLIywz52LtThsYPgiYxY6mvgPe0fd0AY733ljUMqUa8ig==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-5831b8676984d5f93ec19377\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.84\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.711",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.711Z\t21856a47-aa92-46d7-827e-24b4cdfb0e3b\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.712",
        "@message": "END RequestId: 21856a47-aa92-46d7-827e-24b4cdfb0e3b\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.801",
        "@message": "START RequestId: 5b2a0177-81ff-4437-9efc-32235082d651 Version: $LATEST\n"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.802",
        "@message": "[INFO]\t2024-02-02T19:17:44.802Z\t5b2a0177-81ff-4437-9efc-32235082d651\tazul.chalice\tReceived GET request for '/manifest/files/k8Qgp4Qn1udTUw7P7N0js9vLbJLzqHLop1vYzi_Vrwn3_JgAAQ==', with {\"query\": null, \"headers\": {\"accept-encoding\": \"identity\", \"cloudfront-forwarded-proto\": \"https\", \"cloudfront-is-desktop-viewer\": \"true\", \"cloudfront-is-mobile-viewer\": \"false\", \"cloudfront-is-smarttv-viewer\": \"false\", \"cloudfront-is-tablet-viewer\": \"false\", \"cloudfront-viewer-asn\": \"7922\", \"cloudfront-viewer-country\": \"US\", \"host\": \"service.daniel.dev.singlecell.gi.ucsc.edu\", \"user-agent\": \"python-urllib3/1.26.18\", \"via\": \"1.1 2568ed32d67e750311d0391c651da0c6.cloudfront.net (CloudFront)\", \"x-amz-cf-id\": \"X-iw6l7NuTxvbW2i33KdcAjXDZ3-tinmR0O7dwKKU-PI3Z7BAyLvvA==\", \"x-amzn-trace-id\": \"Root=1-65bd3fd8-71be938c3105e75d6eec7d68\", \"x-forwarded-for\": \"73.222.193.39, 70.132.61.153\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\"}}."
    },
    {
        "@timestamp": "2024-02-02 19:17:44.839",
        "@message": "[DEBUG]\t2024-02-02T19:17:44.838Z\t5b2a0177-81ff-4437-9efc-32235082d651\tazul.chalice\tReturning 500 response with headers {\"Access-Control-Allow-Origin\": \"*\", \"Access-Control-Allow-Headers\": \"Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key\", \"Strict-Transport-Security\": \"max-age=31536000; includeSubDomains\"}. See next line for the first 1024 characters of the body.\n{\"Code\": \"ChaliceViewError\", \"Message\": \"('Failed to generate manifest', 'FAILED', None)\"}"
    },
    {
        "@timestamp": "2024-02-02 19:17:44.840",
        "@message": "END RequestId: 5b2a0177-81ff-4437-9efc-32235082d651\n"
    }
]

(No logs for the throttled requests could be found in /aws/lambda/azul-service-daniel-manifest)

azul-service_manifest_throttles-daniel.alarm

Alarm notification sent to azul-group.

@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 3366611 to 15e202f Compare February 5, 2024 18:11
@dsotirho-ucsc dsotirho-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 15e202f to 2e74dd5 Compare February 6, 2024 02:29
@achave11-ucsc achave11-ucsc added the reqs [process] PR includes commit requiring ``make requirements`` label Jul 9, 2025
@achave11-ucsc achave11-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from a83e7ac to 83c6c23 Compare July 9, 2025 18:08
dsotirho-ucsc and others added 17 commits July 14, 2025 15:36
@dsotirho-ucsc @hannes-ucsc
Revert "[h] Disable global download rate limit in prod (#6799)"
f1de2b0 
This reverts commit 71f6547
@dsotirho-ucsc @hannes-ucsc
Revert "Rate limit download requests globally for HCA (#6740)"
52cd41b 
This reverts commit c0eabd5
@dsotirho-ucsc @hannes-ucsc
Rename WAF rules using snake case
31f1986
@dsotirho-ucsc @hannes-ucsc
Update allowed miminum WAF rate limit
1f548fd
@dsotirho-ucsc @hannes-ucsc
Refactor WAF rate rule config variables
22f5a81
@dsotirho-ucsc @hannes-ucsc
Reorder WAF rate rules
4c9dded
@dsotirho-ucsc @hannes-ucsc
Refactor TF data source for NAT Gateway
a4d9bd0
@dsotirho-ucsc @hannes-ucsc
[1/2] Different rate limits depending on HTTP method (#5555)
c29b770
@hannes-ucsc
[2/2] Different rate limits depending on HTTP method (#5555)
cbca5e3
@hannes-ucsc
Convert TypeVar to PEP-695 type parameter
26bee6f
@hannes-ucsc
Convert requirement to assertion woth R
6195892
@hannes-ucsc
Reduce footprint of app.unit_test hack, and generalize it
3cb3b39
@hannes-ucsc
Improve module loading assertions
b6bae49 
Do them sooner and make them less brittle.
@hannes-ucsc
[R] Configure script logging in all templates
e14eb53
@hannes-ucsc
Move function definition
6585687
@achave11-ucsc @hannes-ucsc
[R] Ran make requirements_update
6ae02d1
@hannes-ucsc
[R] Ran make requirements_update
f16a7ea
@hannes-ucsc hannes-ucsc force-pushed the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch from 83c6c23 to f16a7ea Compare July 14, 2025 22:40
hannes-ucsc
hannes-ucsc approved these changes Jul 14, 2025
View reviewed changes
Copy link
Member

@hannes-ucsc hannes-ucsc left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@achave11-ucsc, please run update the requirements and immediately please push this to the GitLabs soon so as to avoid the requirements becoming stale again. The checklists need to be unwound, too. It would be preferable to do that unwinding when the PR is put back from Approved to Sprint.

@hannes-ucsc hannes-ucsc removed the sandbox [process] Resolution is being verified in sandbox deployment label Jul 14, 2025
@achave11-ucsc achave11-ucsc added the sandbox [process] Resolution is being verified in sandbox deployment label Jul 15, 2025
@achave11-ucsc achave11-ucsc merged commit b1bf99d into develop Jul 15, 2025
11 checks passed
@achave11-ucsc achave11-ucsc deleted the issues/dsotirho-ucsc/5555-rate-limit-on-http-method branch July 15, 2025 16:10
@achave11-ucsc achave11-ucsc removed their assignment Jul 15, 2025
@hannes-ucsc hannes-ucsc linked an issue Sep 11, 2025 that may be closed by this pull request
Different rate limits depending on HTTP method #5555
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hannes-ucsc hannes-ucsc hannes-ucsc approved these changes

@achave11-ucsc achave11-ucsc achave11-ucsc left review comments

@nadove-ucsc nadove-ucsc nadove-ucsc left review comments

Assignees

No one assigned

Labels

4+ reviews [process] Lead requested changes four times or more reqs [process] PR includes commit requiring ``make requirements`` sandbox [process] Resolution is being verified in sandbox deployment

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Different rate limits depending on HTTP method

6 participants

@dsotirho-ucsc @coveralls @hannes-ucsc @achave11-ucsc @nadove-ucsc