Skip to content

Transient IT failure due to more than one access method #8132

Open
Defect
#8150
Open
Transient IT failure due to more than one access method#8132
Defect
#8150
Assignees
dsotirho-ucsc
Labels
data[subject] Data or metadata [use of this label is uncommon]spike:2[process] Spike estimate of two points

Description

@dsotirho-ucsc
dsotirho-ucsc
opened on Jun 23, 2026

Transient IT failure on anvilprod

https://gitlab.explore.anvilproject.org/ucsc/azul/-/jobs/68477

2026-06-23T01:00:24.648788Z 01E 2026-06-23 01:00:24,648    INFO MainThread test.integration_test: Beginning sub-test [managed_access_repository_files] {'catalog': 'anvil13-it'}
2026-06-23T01:00:24.649972Z 01E 2026-06-23 01:00:24,649    INFO MainThread azul.oauth2: Making GET request to 'https://service.explore.anvilproject.org/index/files?catalog=anvil13-it&size=100&filters=%7B%22source_id%22%3A+%7B%22is%22%3A+%5B%224729edd0-4fa5-4326-aab1-d105ac236294%22%5D%7D%7D'
2026-06-23T01:00:24.649979Z 01E 2026-06-23 01:00:24,649    INFO MainThread azul.oauth2: … without a request body
2026-06-23T01:00:24.650484Z 01E 2026-06-23 01:00:24,650    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDgg0')]
2026-06-23T01:00:24.824130Z 01E 2026-06-23 01:00:24,823    INFO MainThread azul.oauth2: Got 200 response after 0.174s from GET to https://service.explore.anvilproject.org/index/files?catalog=anvil13-it&size=100&filters=%7B%22source_id%22%3A+%7B%22is%22%3A+%5B%224729edd0-4fa5-4326-aab1-d105ac236294%22%5D%7D%7D
2026-06-23T01:00:24.824140Z 01E 2026-06-23 01:00:24,823    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '67682', 'Connection': 'keep-alive', 'Date': 'Tue, 23 Jun 2026 01:00:24 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a39daa8-11a64e015a060b4345830fad;Parent=7061cb55098009ff;Sampled=0;Lineage=1:45061563:0', 'x-amzn-RequestId': 'e2bbf0df-0504-482a-9365-2d8f716ed6a2', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'fY8aaFzyoAMEPrg=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 1bf9b118a40d760b4b2a9d173dae3efc.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P13', 'X-Amz-Cf-Id': 'DNWEUMfUcBwOnqlOG6hIKMbDurwqPKeGuZcdNwc4Lp_wojPacX3evQ=='})
2026-06-23T01:00:24.824218Z 01E 2026-06-23 01:00:24,823    INFO MainThread azul.oauth2: … with a response body of length 67682 starting in b'{"hits":[{"entryId":"051ad53b-5f59-4550-9038-8fe08c40873b","sources":[{"source_prefix":"2/1","source_spec":"tdr:bigquery:gcp:datarepo-14967a4d:ANVIL_CCDG_WashU_CVD_EOCAD_WashU_CAD_GRU_IRB_WGS_20230525_ANV5_202503171543","source_id":"4729edd0-4fa5-4326-aab1-d105ac236294"}],"bundles":[{"bundle_uuid":"2325c060-1161-a2fc-8fe3-8543a668a62b","bundle_version":"2022-06-01T00:00:00.000000Z"}],"activities":[{"activity_type":["Indexing","Sequencing"],"assay_type":[null],"data_modality":[null]}],"biosamples":[{"anatomical_site":["Unknown"],"biosample_type":["Blood"],"disease":[null],"donor_age_at_collection_unit":["Years"],"donor_age_at_collection":[{"gte":65.0,"lte":65.0}]}],"datasets":[{"dataset_id":["a7506807-fc9d-fd57-7e90-f33bc76779c6"],"consent_group":["GRU-IRB"],"data_use_permission":["GRU-IRB"],"registered_identifier":["phs001227"],"title":["ANVIL_CCDG_WashU_CVD_EOCAD_WashU_CAD_GRU_IRB_WGS"],"data_modality":["whole genome"]}],"diagnoses":[{"disease":["Coronary arteriosclerosis"],"diagnosis_age_unit":[null],"on...'
2026-06-23T01:00:24.837588Z 01E 2026-06-23 01:00:24,837    INFO MainThread test.integration_test: Making GET request to 'https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-06-23T01:00:24.837599Z 01E 2026-06-23 01:00:24,837    INFO MainThread test.integration_test: … without a request body
2026-06-23T01:00:24.837802Z 01E 2026-06-23 01:00:24,837    INFO MainThread test.integration_test: … without request headers
2026-06-23T01:00:24.885889Z 01E 2026-06-23 01:00:24,885    INFO MainThread test.integration_test: Got 404 response after 0.048s from GET to https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-06-23T01:00:24.886095Z 01E 2026-06-23 01:00:24,885    INFO MainThread test.integration_test: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '158', 'Connection': 'keep-alive', 'Date': 'Tue, 23 Jun 2026 01:00:24 GMT', 'x-amzn-RequestId': '81a579f7-d69b-4257-b264-6257b890c2d1', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'fY8acFYYoAMEYSQ=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Amzn-Trace-Id': 'Root=1-6a39daa8-53fbd2431e4a4a2f03597b5e;Parent=5dfd276862f8735a;Sampled=0;Lineage=1:45061563:0', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 4e65e40699a5e85cdc5b08cf225325c4.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P13', 'X-Amz-Cf-Id': 'TMcsQyec9UYNR-wFPB5pEIRMe7wGu4jH-1hqXWJt0MQKbvZGbboehA=='})
2026-06-23T01:00:24.886106Z 01E 2026-06-23 01:00:24,885    INFO MainThread test.integration_test: … with a response body of length 158 being b'{"Code":"NotFoundError","Message":"Unable to find file \'e56debc7-8c57-42f3-b4f3-57df891097ea\', version \'2022-06-01T00:00:00.000000Z\' in catalog \'anvil13-it\'"}'
2026-06-23T01:00:24.886890Z 01E 2026-06-23 01:00:24,886    INFO MainThread azul.oauth2: Making GET request to 'https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-06-23T01:00:24.886896Z 01E 2026-06-23 01:00:24,886    INFO MainThread azul.oauth2: … without a request body
2026-06-23T01:00:24.887363Z 01E 2026-06-23 01:00:24,887    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDgg0')]
2026-06-23T01:00:27.610324Z 01E 2026-06-23 01:00:27,609    INFO MainThread azul.oauth2: Got 500 response after 2.723s from GET to https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-06-23T01:00:27.610668Z 01E 2026-06-23 01:00:27,609    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Tue, 23 Jun 2026 01:00:27 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a39daa8-3c4578b62c317ebb5d7a4662;Parent=54e47b4bc0c2a080;Sampled=0;Lineage=1:45061563:0', 'x-amzn-RequestId': '0103d376-4cb9-4926-8455-e05251d0fd51', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'fY8acH6SoAMEA9A=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 1bf9b118a40d760b4b2a9d173dae3efc.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P13', 'X-Amz-Cf-Id': 'dbflAEloQmZMR2cIcC07GTlNzvT6qXkBoF6QzQMbSE5TwIerWDv2TQ=='})
2026-06-23T01:00:27.610679Z 01E 2026-06-23 01:00:27,610    INFO MainThread azul.oauth2: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"5f212470-99a4-48c7-8bab-5b874e1287e6"}'
2026-06-23T01:00:27.610682Z 01E 2026-06-23 01:00:27,610    INFO MainThread azul.oauth2: Making GET request to 'https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-06-23T01:00:27.610685Z 01E 2026-06-23 01:00:27,610    INFO MainThread azul.oauth2: … without a request body
2026-06-23T01:00:27.611081Z 01E 2026-06-23 01:00:27,610    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDgg0')]
2026-06-23T01:00:28.693453Z 01E 2026-06-23 01:00:28,692    INFO MainThread azul.oauth2: Got 500 response after 1.082s from GET to https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-06-23T01:00:28.693464Z 01E 2026-06-23 01:00:28,692    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Tue, 23 Jun 2026 01:00:28 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a39daab-1905b71f4ca299306984a3fd;Parent=18f839eff1a3a96d;Sampled=0;Lineage=1:45061563:0', 'x-amzn-RequestId': 'c085b6e9-5b8a-40fd-8184-4d88cab68d17', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'fY8a4GSNIAMESVQ=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 1bf9b118a40d760b4b2a9d173dae3efc.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P13', 'X-Amz-Cf-Id': 'YvvRtnIzm0B9KNmy9loViAaRlYhrTh_tsr4rrHaEr5eTblOUUJM-oA=='})
2026-06-23T01:00:28.693515Z 01E 2026-06-23 01:00:28,693    INFO MainThread azul.oauth2: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"f540987c-3cd2-436a-b5d8-bbc6e46bb5ae"}'
2026-06-23T01:00:28.693609Z 01E 2026-06-23 01:00:28,693    INFO MainThread azul.oauth2: Making GET request to 'https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-06-23T01:00:28.693613Z 01E 2026-06-23 01:00:28,693    INFO MainThread azul.oauth2: … without a request body
2026-06-23T01:00:28.694053Z 01E 2026-06-23 01:00:28,693    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDgg0')]
2026-06-23T01:00:29.630735Z 01E 2026-06-23 01:00:29,630    INFO MainThread azul.oauth2: Got 500 response after 0.937s from GET to https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-06-23T01:00:29.630946Z 01E 2026-06-23 01:00:29,630    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Tue, 23 Jun 2026 01:00:29 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a39daac-1340858122aead125498eb6b;Parent=4f0b0d2cc8f3ce1f;Sampled=0;Lineage=1:45061563:0', 'x-amzn-RequestId': '57918b0b-64b7-448e-b780-c9c5d728537a', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'fY8bCFtdIAMEXkA=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 1bf9b118a40d760b4b2a9d173dae3efc.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P13', 'X-Amz-Cf-Id': 'RDSVVv6aK5M9AoXCnpehJ1fUekOIjzlY9Lo7QjArvtaSMSPjDG1ywg=='})
2026-06-23T01:00:29.630976Z 01E 2026-06-23 01:00:29,630    INFO MainThread azul.oauth2: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"fe0e3195-847b-4d6d-a51b-43060ed341af"}'
2026-06-23T01:00:29.630979Z 01E 2026-06-23 01:00:29,630    INFO MainThread azul.oauth2: Making GET request to 'https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-06-23T01:00:29.630982Z 01E 2026-06-23 01:00:29,630    INFO MainThread azul.oauth2: … without a request body
2026-06-23T01:00:29.631328Z 01E 2026-06-23 01:00:29,631    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDgg0')]
2026-06-23T01:00:32.213803Z 01E 2026-06-23 01:00:32,213    INFO MainThread azul.oauth2: Got 500 response after 2.583s from GET to https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-06-23T01:00:32.213976Z 01E 2026-06-23 01:00:32,213    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Tue, 23 Jun 2026 01:00:32 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a39daad-10c30c8d1ec7f6a8659e41ee;Parent=109fbadd27e989b2;Sampled=0;Lineage=1:45061563:0', 'x-amzn-RequestId': '0477abc8-95a8-4c64-a0af-071ea56afc36', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'fY8bMF1cIAMEvRQ=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 1bf9b118a40d760b4b2a9d173dae3efc.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P13', 'X-Amz-Cf-Id': 'QRxbqV9Fe7nfqD5c5Edjibu917M5fg5yckGcv_dKIS-c0QWmAFgAVA=='})
2026-06-23T01:00:32.214062Z 01E 2026-06-23 01:00:32,213    INFO MainThread azul.oauth2: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"16988514-12b3-4595-9e54-27c55cbf9133"}'
2026-06-23T01:00:32.214234Z 01E 2026-06-23 01:00:32,213    INFO MainThread azul.oauth2: Making GET request to 'https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-06-23T01:00:32.214238Z 01E 2026-06-23 01:00:32,213    INFO MainThread azul.oauth2: … without a request body
2026-06-23T01:00:32.214715Z 01E 2026-06-23 01:00:32,214    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDgg0')]
2026-06-23T01:00:33.325554Z 01E 2026-06-23 01:00:33,325    INFO MainThread azul.oauth2: Got 500 response after 1.111s from GET to https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-06-23T01:00:33.325565Z 01E 2026-06-23 01:00:33,325    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Tue, 23 Jun 2026 01:00:33 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a39dab0-6c4cf89674c1c0223bc3f6da;Parent=75e2a48c8aa81ae3;Sampled=0;Lineage=1:45061563:0', 'x-amzn-RequestId': '712833bb-d952-4d72-b650-bfd8b5f9b1bc', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'fY8bmH2XIAMEFYQ=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 1bf9b118a40d760b4b2a9d173dae3efc.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P13', 'X-Amz-Cf-Id': 'PInGUxIWZgxt5eLfn75CzyBZYm9QG53ibTh13g8l2XzMzkJwWfa8pA=='})
2026-06-23T01:00:33.325889Z 01E 2026-06-23 01:00:33,325    INFO MainThread azul.oauth2: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"c5250c10-08ff-43bb-a639-46527c51a843"}'
2026-06-23T01:00:33.325895Z 01E 2026-06-23 01:00:33,325    INFO MainThread azul.oauth2: Making GET request to 'https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z'
2026-06-23T01:00:33.325898Z 01E 2026-06-23 01:00:33,325    INFO MainThread azul.oauth2: … without a request body
2026-06-23T01:00:33.326419Z 01E 2026-06-23 01:00:33,326    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'Bearer ya2REDACTEDgg0')]
2026-06-23T01:00:35.516445Z 01E 2026-06-23 01:00:35,515    INFO MainThread azul.oauth2: Got 500 response after 2.190s from GET to https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z
2026-06-23T01:00:35.516813Z 01E 2026-06-23 01:00:35,516    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '128', 'Connection': 'keep-alive', 'Date': 'Tue, 23 Jun 2026 01:00:35 GMT', 'X-Amzn-Trace-Id': 'Root=1-6a39dab1-577e4955797dcfa962aae6f4;Parent=2bbef21dcd820fa4;Sampled=0;Lineage=1:45061563:0', 'x-amzn-RequestId': '1a4679e7-4d47-4937-8351-7c25e1b34d68', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'fY8bxF-cIAMEQQg=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 1bf9b118a40d760b4b2a9d173dae3efc.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P13', 'X-Amz-Cf-Id': 'Df0N8exCdPdTeZ_TVmiwcCbvWdOTRS91guJUnnW3IofAl_2ggrLEQA=='})
2026-06-23T01:00:35.516826Z 01E 2026-06-23 01:00:35,516    INFO MainThread azul.oauth2: … with a response body of length 128 being b'{"Code":"InternalServerError","Message":"An internal server error occurred.","RequestId":"88a5c866-cab1-46b9-b496-6c14aac3c7a5"}'
2026-06-23T01:00:35.516829Z 01E 2026-06-23 01:00:35,516    INFO MainThread test.integration_test: Failed sub-test [managed_access_repository_files] {'catalog': 'anvil13-it'}
...
2026-06-23T01:07:51.125266Z 01E ======================================================================
2026-06-23T01:07:51.125268Z 01E ERROR: test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [managed_access_repository_files] (catalog='anvil13-it')
2026-06-23T01:07:51.125422Z 01E ----------------------------------------------------------------------
2026-06-23T01:07:51.125806Z 01E Traceback (most recent call last):
2026-06-23T01:07:51.125814Z 01E   File "/builds/ucsc/azul/test/integration_test.py", line 433, in subTest
2026-06-23T01:07:51.125816Z 01E     yield
2026-06-23T01:07:51.125818Z 01E   File "/builds/ucsc/azul/test/integration_test.py", line 1641, in _test_managed_access
2026-06-23T01:07:51.125820Z 01E     files = self._test_managed_access_repository_files(catalog, ma_source)
2026-06-23T01:07:51.125822Z 01E   File "/builds/ucsc/azul/test/integration_test.py", line 1778, in _test_managed_access_repository_files
2026-06-23T01:07:51.125824Z 01E     response = self._get_url_unchecked(GET, file_url)
2026-06-23T01:07:51.125826Z 01E   File "/builds/ucsc/azul/test/integration_test.py", line 1169, in _get_url_unchecked
2026-06-23T01:07:51.125829Z 01E     response = http.request(method=method,
2026-06-23T01:07:51.125830Z 01E                             url=url,
2026-06-23T01:07:51.125832Z 01E     ...<2 lines>...
2026-06-23T01:07:51.125834Z 01E                             timeout=float(config.api_gateway_lambda_timeout + 1),
2026-06-23T01:07:51.125837Z 01E                             preload_content=not stream)
2026-06-23T01:07:51.125840Z 01E   File "/build/.venv/lib/python3.14/site-packages/urllib3/_request_methods.py", line 135, in request
2026-06-23T01:07:51.125842Z 01E     return self.request_encode_url(
2026-06-23T01:07:51.125846Z 01E            ~~~~~~~~~~~~~~~~~~~~~~~^
2026-06-23T01:07:51.125850Z 01E         method,
2026-06-23T01:07:51.125853Z 01E         ^^^^^^^
2026-06-23T01:07:51.125856Z 01E     ...<3 lines>...
2026-06-23T01:07:51.125858Z 01E         **urlopen_kw,
2026-06-23T01:07:51.125859Z 01E         ^^^^^^^^^^^^^
2026-06-23T01:07:51.125860Z 01E     )
2026-06-23T01:07:51.125861Z 01E     ^
2026-06-23T01:07:51.125862Z 01E   File "/build/.venv/lib/python3.14/site-packages/urllib3/_request_methods.py", line 182, in request_encode_url
2026-06-23T01:07:51.125863Z 01E     return self.urlopen(method, url, **extra_kw)
2026-06-23T01:07:51.125864Z 01E            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
2026-06-23T01:07:51.125866Z 01E   File "/build/.venv/lib/python3.14/site-packages/google/auth/transport/urllib3.py", line 401, in urlopen
2026-06-23T01:07:51.125867Z 01E     response = self.http.urlopen(
2026-06-23T01:07:51.125868Z 01E         method, url, body=body, headers=request_headers, **kwargs
2026-06-23T01:07:51.125869Z 01E     )
2026-06-23T01:07:51.125870Z 01E   File "/builds/ucsc/azul/src/azul/http.py", line 53, in urlopen
2026-06-23T01:07:51.125899Z 01E     return self._inner.urlopen(*args, **kwargs)
2026-06-23T01:07:51.125900Z 01E            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
2026-06-23T01:07:51.125901Z 01E   File "/builds/ucsc/azul/src/azul/http.py", line 458, in urlopen
2026-06-23T01:07:51.125902Z 01E     raise urllib3.exceptions.MaxRetryError(pool, url)
2026-06-23T01:07:51.125904Z 01E urllib3.exceptions.MaxRetryError: azul.oauth2.LoggingHTTPSConnectionPool(host='service.explore.anvilproject.org', port=443): Max retries exceeded with url: https://service.explore.anvilproject.org/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea?catalog=anvil13-it&version=2022-06-01T00%3A00%3A00.000000Z (Caused by None)
2026-06-23T01:07:51.125907Z 01E

CloudWatch Log Analytics Results
Region: us-east-1
Query:

SOURCE "/aws/lambda/azul-service-anvilprod" START=-43200s END=0s |
fields @timestamp, @message
| filter @message like /ERROR|Status: timeout|5f212470/
| sort @timestamp asc
| limit 1000
@timestamp @message
2026-06-23 01:00:24.910 START RequestId: 5f212470-99a4-48c7-8bab-5b874e1287e6 Version: 40
2026-06-23 01:00:24.911 [INFO] 2026-06-23T01:00:24.911Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.chalice Authenticated request as AccessTokenAuthentication(token='ya29.c.c0AZ-REDACTED')
2026-06-23 01:00:24.911 [INFO] 2026-06-23T01:00:24.911Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.chalice Received GET request for '/repository/files/e56debc7-8c57-42f3-b4f3-57df891097ea', with {"query": {"catalog": "anvil13-it", "version": "2022-06-01T00:00:00.000000Z"}, "headers": {"accept-encoding": "identity", "authorization": "Bearer ya29.c.c0AZ-REDACTED", "cloudfront-forwarded-proto": "https", "cloudfront-is-desktop-viewer": "true", "cloudfront-is-mobile-viewer": "false", "cloudfront-is-smarttv-viewer": "false", "cloudfront-is-tablet-viewer": "false", "cloudfront-viewer-asn": "14618", "cloudfront-viewer-country": "US", "host": "service.explore.anvilproject.org", "user-agent": "python-urllib3/2.7.0", "via": "1.1 1bf9b118a40d760b4b2a9d173dae3efc.cloudfront.net (CloudFront)", "x-amz-cf-id": "dbflAEloQmZMR2cIcC07GTlNzvT6qXkBoF6QzQMbSE5TwIerWDv2TQ==", "x-amzn-trace-id": "Root=1-6a39daa8-3c4578b62c317ebb5d7a4662", "x-forwarded-for": "35.168.152.160, 15.158.251.79", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-goog-api-client": "cred-type/sa"}}.
2026-06-23 01:00:24.911 [INFO] 2026-06-23T01:00:24.911Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.chalice … without a request body
2026-06-23 01:00:24.912 [INFO] 2026-06-23T01:00:24.912Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.boto3 dynamodb.GetItem: Making POST request to https://dynamodb.us-east-1.amazonaws.com/
2026-06-23 01:00:24.913 [INFO] 2026-06-23T01:00:24.913Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.boto3 dynamodb.GetItem: … with a request body of length 1166 starting in b'{"TableName": "azul-sources_cache_by_auth-anvilprod", "Key": {"identity": {"S": "anvil13-it:ya29.c.c0AZ-REDACTED'
2026-06-23 01:00:24.918 [INFO] 2026-06-23T01:00:24.918Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.boto3 dynamodb.GetItem: Got 200 response
2026-06-23 01:00:24.919 [INFO] 2026-06-23T01:00:24.919Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.boto3 dynamodb.GetItem: … with a response body of length 96409 starting in b'{"Item":{"expiration":{"N":"1782176654"},"sources":{"S":"[\"29f7d10b-778e-4cf1-978c-c3893d0f2bc0\", \"7117ca43-5946-4895-90f7-779f2bb216ae\", \"973a7eca-4376-4997-becf-24e61f8a3b62\", \"e57ce716-664c-49dd-9e43-98fa5b163f73\", \"494faa66-e123-4b1f-860c-4cefbc62b3ae\", \"634d44c2-f101-44dd-9568-0ea2e080855d\", \"4886d9b5-4dac-433d-a5d5-323b5cf21425\", \"b8f39bec-0137-485b-a6f2-e8c5cf8e80e8\", \"a31265a3-b65a-42de-ad6c-0380238e569f\", \"7920973b-0882-4746-813b-cd8e3da8184e\", \"b70d79e2-771d-4fb0-95ef-bc38d93aff1d\", \"5595525c-b73d-4186-a352-18afeb89ef62\", \"e48ebc69-4481-49ab-a107-3836012755d8\", \"dee17f6d-8c5c-4f0f-b692-1277be521c91\", \"4ec26ff6-ff6e-4f71-bfbe-e2bc49d2d6a9\", \"28317b55-5156-4908-8d7f-4a695ff1bf41\", \"feeb060c-87db-47ab-a960-aa308f914021\", \"592d2b12-ea32-4baa-81e0-4cacf7ff4a3e\", \"4addf51d-b7f3-43fa-bc27-9649d4a0f9c2\", \"3ccf79bc-89b8-4448-96e9-605e05d0e7b4\", \"598c1295-74c2-4a7e-930e-f46722d6750a\", \"7a6e5789-184f-45a9-9be4-a118cb11b083\", \"d569b59d-f7fa-40a6-9200-829aeb869bc3\...'
2026-06-23 01:00:24.935 [INFO] 2026-06-23T01:00:24.935Z 5f212470-99a4-48c7-8bab-5b874e1287e6 opensearch Making POST request to https://vpc-azul-index-anvilprod-ggipah4skn2ftt47u4xgvydzqm.us-east-1.es.amazonaws.com:443/azul_v2_anvilprod_anvil13-it_files_aggregate/_search
2026-06-23 01:00:24.936 [INFO] 2026-06-23T01:00:24.936Z 5f212470-99a4-48c7-8bab-5b874e1287e6 opensearch … with a request body of length 89787 starting in b'{"query":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.files.document_id.keyword":["e56debc7-8c57-42f3-b4f3-57df891097ea"]}}}},{"constant_score":{"filter":{"terms":{"contents.files.version.keyword":["2022-06-01T00:00:00.000000Z"]}}}},{"constant_score":{"filter":{"terms":{"sources.id.keyword":["00002787-81de-47e4-a7a9-7da09bc95592","004ee0a4-7672-4922-9ec2-9f1c4e1ff4a9","0070f6a4-5430-47aa-b1fb-f7d175a470c8","008578bf-7e98-455f-8c11-75db70cb7fac","009e77b1-362e-47e0-9ae7-a03a390a07ce","00c1a1f1-6bfc-478b-8b32-5d5911081638","00d1422e-2055-4a74-af2e-c7ab1e25cd5e","010b4ea5-9dd4-4e2b-add8-8e69f7c509f4","01795dd6-5123-4ed4-88de-06a1a99f75af","01a1e8d3-81f5-4680-bcfb-083527e71982","01a8a72d-0a21-4bbf-b50f-6f1687cf1e61","01cf2450-604b-43e5-9f4e-9ec4e0bf0a61","01d0e707-2021-4740-9b21-60c8ad24fc76","01e26559-f5a5-42f8-b601-c24ef42f4aba","01ebc9c6-08de-41e5-a689-aaa5685a1d92","020e1f1a-7314-44b5-a281-942a45269815","02137a6c-e68a-4ccb-b5aa-e1013d8e44da","023486a4-2a09-4a19-aba4-f717a7483e6e","02376...'
2026-06-23 01:00:24.959 [INFO] 2026-06-23T01:00:24.959Z 5f212470-99a4-48c7-8bab-5b874e1287e6 opensearch Got 200 response after 0.023s from POST to https://vpc-azul-index-anvilprod-ggipah4skn2ftt47u4xgvydzqm.us-east-1.es.amazonaws.com:443/azul_v2_anvilprod_anvil13-it_files_aggregate/_search
2026-06-23 01:00:24.959 [INFO] 2026-06-23T01:00:24.959Z 5f212470-99a4-48c7-8bab-5b874e1287e6 opensearch … with a response body of length 3054 starting in '{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":3.0,"hits":[{"_index":"azul_v2_anvilprod_anvil13-it_files_aggregate","_id":"e56debc7-8c57-42f3-b4f3-57df891097ea","_score":3.0,"_source":{"entity_id":"e56debc7-8c57-42f3-b4f3-57df891097ea","contents":{"activities":[{"document_id":["5a2a36e9-77f7-4152-b9ed-f0b186b1256c","952a7ed7-6c9b-443b-bdbf-804d48b70a75"],"source_datarepo_row_ids":["file_inventory:a22c3d44-4416-4cd2-8cea-b4d82cf21503","file_inventory:adc87ab8-c036-4383-a791-0602ab5e3514","sequencing:5e479f69-dfa1-4186-9c74-3acde7b7fb08"],"activity_id":["65d8322d-ac45-0f0e-e36e-2e7a4160f72f","d0e920a7-9557-fc46-fa5e-bc6e2d65c898"],"activity_table":["anvil_activity","anvil_sequencingactivity"],"activity_type":["Indexing","Sequencing"],"assay_type":["~null"],"data_modality":["~null"],"reference_assembly":["~null"]}],"biosamples":[{"document_id":["21301bac-f0d4-409d-9cd9-f813f0abae76"],"source_datarepo_row_ids":["samp…'
2026-06-23 01:00:24.962 [INFO] 2026-06-23T01:00:24.962Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.boto3 s3.HeadObject: Making HEAD request to https://s3.amazonaws.com/edu-ucsc-gi-platform-anvil-prod-mamirror-anvilprod.us-east-1/_it/info/6dba00b6865e79024cde852556c1cb6f.md5.json
2026-06-23 01:00:24.962 [INFO] 2026-06-23T01:00:24.962Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.boto3 s3.HeadObject: … without a request body
2026-06-23 01:00:25.031 [INFO] 2026-06-23T01:00:25.031Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.drs Obtained URL pattern 'https://data.terra.bio/ga4gh/drs/v1/objects/{$id}' from resource 'National Human Genome Research Institute'
2026-06-23 01:00:25.031 [INFO] 2026-06-23T01:00:25.031Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.drs Resolved prefix 'drs.anv0' to namespace ID '3571'
2026-06-23 01:00:25.031 [INFO] 2026-06-23T01:00:25.031Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.boto3 s3.HeadObject: Got 404 response
2026-06-23 01:00:25.031 [INFO] 2026-06-23T01:00:25.031Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.boto3 s3.HeadObject: … without a response body
2026-06-23 01:00:25.032 [INFO] 2026-06-23T01:00:25.032Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.terra Making GET request to 'https://data.terra.bio/ga4gh/drs/v1/objects/v2_47d05b81-81d5-3397-bb48-2b0d9721c727'
2026-06-23 01:00:25.032 [INFO] 2026-06-23T01:00:25.032Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.terra … with request headers [('x-goog-api-client', 'cred-type/u'), ('authorization', 'Bearer ya2REDACTEDgg0')]
2026-06-23 01:00:25.032 [INFO] 2026-06-23T01:00:25.032Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.terra … without a request body
2026-06-23 01:00:27.597 [INFO] 2026-06-23T01:00:27.597Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.terra … with response headers HTTPHeaderDict({'Date': 'Tue, 23 Jun 2026 01:00:27 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'awWJNOvy', 'Content-Type': 'application/json', 'Content-Length': '4432', 'Set-Cookie': 'JSESSIONID=5D47197E4584D08459EA756AC5A39117; Path=/; Secure; HttpOnly; SameSite=Lax', 'Vary': 'Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000'})
2026-06-23 01:00:27.597 [INFO] 2026-06-23T01:00:27.597Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.terra Got 200 response after 2.565s from GET to https://data.terra.bio/ga4gh/drs/v1/objects/v2_47d05b81-81d5-3397-bb48-2b0d9721c727
2026-06-23 01:00:27.597 [INFO] 2026-06-23T01:00:27.597Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.terra … with a response body of length 4432 starting in b'{"id":"v2_47d05b81-81d5-3397-bb48-2b0d9721c727","name":"H_VL-WU-05165-AB65717830_H_VL-WU-05165-AB65717830.cram.crai","self_uri":"drs://data.terra.bio/v2_47d05b81-81d5-3397-bb48-2b0d9721c727","size":936045,"created_time":"2022-08-26T15:56:11.679Z","updated_time":"2022-08-26T15:56:23.345Z","version":"0","mime_type":"application/octet-stream","checksums":[{"checksum":"71c16737","type":"crc32c"},{"checksum":"6dba00b6865e79024cde852556c1cb6f","type":"md5"}],"access_methods":[{"type":"gs","access_url":{"url":"gs://fc-secure-1a841909-2e43-4c83-8b66-ba15aceecfee/H_VL-WU-05165-AB65717830/H_VL-WU-05165-AB65717830.cram.crai","headers":null},"access_id":"gcp-us-central1*f7307ad5-7250-483c-a8e2-aeb1c1226b13","region":"us-central1","authorizations":{"supported_types":["BearerAuth"],"passport_auth_issuers":null,"bearer_auth_issuers":null}},{"type":"gs","access_url":{"url":"gs://fc-secure-87fff435-fd01-4bd9-91b4-8670f7b19162/H_VL-WU-05165-AB65717830/H_VL-WU-05165-AB65717830.cram.crai","headers":null},"access_id":"gcp-us-c...'
2026-06-23 01:00:27.602 [ERROR] 2026-06-23T01:00:27.598Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul-service-anvilprod Caught exception for path /repository/files/{file_uuid} Traceback (most recent call last): File "/var/task/chalice/app.py", line 1922, in _get_view_function_response response = view_function(**function_args) File "/var/task/azul/service/repository_controller.py", line 231, in get_repository_files result = self.download_file(file_uuid, fetch=False) File "/var/task/azul/service/repository_controller.py", line 334, in download_file return self._download_file(catalog=catalog, ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ fetch=fetch, ^^^^^^^^^^^^ ...<2 lines>... headers=headers, ^^^^^^^^^^^^^^^^ authentication=authentication) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/service/repository_controller.py", line 436, in _download_file download.update(authentication) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ File "/var/task/azul/plugins/repository/tdr.py", line 291, in update access = drs_client.get(access_method=AccessMethod.gs, headers=headers) File "/var/task/azul/drs.py", line 377, in get return self._get(access_method, headers) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/drs.py", line 390, in _get method = one(m for m in access_methods if m['type'] == access_method.scheme) File "/opt/python/more_itertools/more.py", line 661, in one raise too_long or ValueError(msg) ValueError: Expected exactly one item in iterable, but got {'type': 'gs', 'access_url': {'url': 'gs://fc-secure-1a841909-2e43-4c83-8b66-ba15aceecfee/H_VL-WU-05165-AB65717830/H_VL-WU-05165-AB65717830.cram.crai', 'headers': None}, 'access_id': 'gcp-us-central1f7307ad5-7250-483c-a8e2-aeb1c1226b13', 'region': 'us-central1', 'authorizations': {'supported_types': ['BearerAuth'], 'passport_auth_issuers': None, 'bearer_auth_issuers': None}}, {'type': 'gs', 'access_url': {'url': 'gs://fc-secure-87fff435-fd01-4bd9-91b4-8670f7b19162/H_VL-WU-05165-AB65717830/H_VL-WU-05165-AB65717830.cram.crai', 'headers': None}, 'access_id': 'gcp-us-central1f7307ad5-7250-483c-a8e2-aeb1c1226b13', 'region': 'us-central1', 'authorizations': {'supported_types': ['BearerAuth'], 'passport_auth_issuers': None, 'bearer_auth_issuers': None}}, and perhaps more.
2026-06-23 01:00:27.602 [INFO] 2026-06-23T01:00:27.602Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.chalice … with a response body of length 133 being {"Code": "InternalServerError", "Message": "An internal server error occurred.", "RequestId": "5f212470-99a4-48c7-8bab-5b874e1287e6"}
2026-06-23 01:00:27.602 [INFO] 2026-06-23T01:00:27.602Z 5f212470-99a4-48c7-8bab-5b874e1287e6 azul.chalice Returning 500 response with headers {"headers": {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}}.
2026-06-23 01:00:27.605 END RequestId: 5f212470-99a4-48c7-8bab-5b874e1287e6
2026-06-23 01:00:27.605 REPORT RequestId: 5f212470-99a4-48c7-8bab-5b874e1287e6 Duration: 2693.91 ms Billed Duration: 2694 ms Memory Size: 768 MB Max Memory Used: 198 MB 
Traceback (most recent call last):
  File "/var/task/chalice/app.py", line 1922, in _get_view_function_response
    response = view_function(**function_args)
  File "/var/task/azul/service/repository_controller.py", line 231, in get_repository_files
    result = self.download_file(file_uuid, fetch=False)
  File "/var/task/azul/service/repository_controller.py", line 334, in download_file
    return self._download_file(catalog=catalog,
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                               fetch=fetch,
                               ^^^^^^^^^^^^
    ...<2 lines>...
                               headers=headers,
                               ^^^^^^^^^^^^^^^^
                               authentication=authentication)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/azul/service/repository_controller.py", line 436, in _download_file
    download.update(authentication)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/var/task/azul/plugins/repository/tdr.py", line 291, in update
    access = drs_client.get(access_method=AccessMethod.gs, headers=headers)
  File "/var/task/azul/drs.py", line 377, in get
    return self._get(access_method, headers)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/task/azul/drs.py", line 390, in _get
    method = one(m for m in access_methods if m['type'] == access_method.scheme)
  File "/opt/python/more_itertools/more.py", line 661, in one
    raise too_long or ValueError(msg)
ValueError: Expected exactly one item in iterable, but got {'type': 'gs', 'access_url': {'url': 'gs://fc-secure-1a841909-2e43-4c83-8b66-ba15aceecfee/H_VL-WU-05165-AB65717830/H_VL-WU-05165-AB65717830.cram.crai', 'headers': None}, 'access_id': 'gcp-us-central1*f7307ad5-7250-483c-a8e2-aeb1c1226b13', 'region': 'us-central1', 'authorizations': {'supported_types': ['BearerAuth'], 'passport_auth_issuers': None, 'bearer_auth_issuers': None}}, {'type': 'gs', 'access_url': {'url': 'gs://fc-secure-87fff435-fd01-4bd9-91b4-8670f7b19162/H_VL-WU-05165-AB65717830/H_VL-WU-05165-AB65717830.cram.crai', 'headers': None}, 'access_id': 'gcp-us-central1*f7307ad5-7250-483c-a8e2-aeb1c1226b13', 'region': 'us-central1', 'authorizations': {'supported_types': ['BearerAuth'], 'passport_auth_issuers': None, 'bearer_auth_issuers': None}}, and perhaps more.

Metadata

Metadata

Assignees

Labels

data[subject] Data or metadata [use of this label is uncommon]spike:2[process] Spike estimate of two points

Type

Defect

Fields

No fields configured for Defect.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions