Remove `dcp-type` MIME parameter

[nadove-ucsc]

Azul appends "; dcp-type=data" to the content-type column of HCA files, presumable to emulate the response from DSS as closely as possible. This is not useful and complicates the verification of the files' content types when mirroring.

[hannes-ucsc]

There's a complication to this that's affecting prod right now: some descriptors in DCP/2 already contain the parameter, leading to a 500 response from the /repository/files endpoint once the file has been mirrorred. For example, attempting to download the biggest file in https://explore.data.humancellatlas.org/projects/60ea42e1-af49-42f5-8164-d641fdb696bc yields the following logs

CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-service-prod
start-time: -3600s
end-time: 0s
query-string:

fields @timestamp, @message
| filter @requestId = '04fe279a-7fa9-4f86-a3f7-06111452af13'
| sort @timestamp desc
| limit 10000

---

@timestamp	@message
2025-05-19 00:06:09.801	END RequestId: 04fe279a-7fa9-4f86-a3f7-06111452af13
2025-05-19 00:06:09.801	REPORT RequestId: 04fe279a-7fa9-4f86-a3f7-06111452af13 Duration: 106.40 ms Billed Duration: 107 ms Memory Size: 2048 MB Max Memory Used: 239 MB
2025-05-19 00:06:09.800	[DEBUG] 2025-05-19T00:06:09.800Z 04fe279a-7fa9-4f86-a3f7-06111452af13 azul.chalice Returning 500 response with headers {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}. See next line for the first 1024 characters of the body. {"Code": "InternalServerError", "Message": "An internal server error occurred.", "RequestId": "04fe279a-7fa9-4f86-a3f7-06111452af13"}
2025-05-19 00:06:09.799	[ERROR] 2025-05-19T00:06:09.798Z 04fe279a-7fa9-4f86-a3f7-06111452af13 azul-service-prod Caught exception for path /fetch/repository/files/{file_uuid} Traceback (most recent call last): File "/var/task/chalice/app.py", line 1922, in _get_view_function_response response = view_function(**function_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/app.py", line 1591, in fetch_repository_files body = _repository_files(file_uuid, fetch=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/app.py", line 1605, in _repository_files return app.repository_controller.download_file(catalog=catalog, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/service/repository_controller.py", line 268, in download_file is_mirrored = self.mirror_service.is_mirrored(catalog, file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/indexer/mirror_service.py", line 192, in is_mirrored return self._get_info(catalog, file) is not None ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/task/azul/indexer/mirror_service.py", line 181, in _get_info assert content_type == file.content_type, R( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: R('Conflicting content type', 'binary/octet-stream; dcp-type=data', HCAFile(uuid='674fc0f9-0da1-42a6-b8f7-08b81a4b3a54', name='H_SN3a_B16_Healthy_SN_1cell_BN1_S832_S448_R2_001.fastq.gz', version='2021-05-19T16:59:22.841000Z', drs_uri='drs://data.terra.bio/v1_f2055f78-5390-4346-9386-32911c638509_531140ad-e9e9-4bae-8e74-185653971882', size=1275537758, content_type='binary/octet-stream', sha256='a569c71966977b558183959a2628222e2a1caba9da16e079d5bc7e60a0d9a9cf', crc32c='17fd0705', sha1=None, s3_etag=None))
2025-05-19 00:06:09.797	[INFO] 2025-05-19T00:06:09.797Z 04fe279a-7fa9-4f86-a3f7-06111452af13 azul.boto3 s3.GetObject: Got 200 response
2025-05-19 00:06:09.797	[INFO] 2025-05-19T00:06:09.797Z 04fe279a-7fa9-4f86-a3f7-06111452af13 azul.boto3 s3.GetObject: … with nonprintable body (<class 'botocore.response.StreamingBody'>)
2025-05-19 00:06:09.733	[INFO] 2025-05-19T00:06:09.733Z 04fe279a-7fa9-4f86-a3f7-06111452af13 azul.boto3 s3.GetObject: Making GET request to https://humancellatlas.s3.amazonaws.com/info/a569c71966977b558183959a2628222e2a1caba9da16e079d5bc7e60a0d9a9cf.sha256.json
2025-05-19 00:06:09.733	[INFO] 2025-05-19T00:06:09.733Z 04fe279a-7fa9-4f86-a3f7-06111452af13 azul.boto3 s3.GetObject: … without request body
2025-05-19 00:06:09.731	[INFO] 2025-05-19T00:06:09.730Z 04fe279a-7fa9-4f86-a3f7-06111452af13 elasticsearch Got 200 response after 0.019s from POST to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp48_files_aggregate/_search
2025-05-19 00:06:09.731	[INFO] 2025-05-19T00:06:09.731Z 04fe279a-7fa9-4f86-a3f7-06111452af13 elasticsearch … with response body '{"took":6,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"…'
2025-05-19 00:06:09.712	[INFO] 2025-05-19T00:06:09.712Z 04fe279a-7fa9-4f86-a3f7-06111452af13 elasticsearch … with request body b'{"query":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.files.uuid.keyword":["674fc0f9-0da1-42a6-b8f7-08b81...'
2025-05-19 00:06:09.711	[INFO] 2025-05-19T00:06:09.711Z 04fe279a-7fa9-4f86-a3f7-06111452af13 elasticsearch Making POST request to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp48_files_aggregate/_search

---

Screenshot showing that the parameter is present in the TDR snapshot

Screenshot showing the fact that in the ES index, the parameter erroneously occurs twice in the content type field:

[hannes-ucsc]

I will add a hotfix to PROD that disables the assertion.

Spike to determine how many file entities in TDR are afflicted by this issue. We may need to fix the metadata, in a separate issue.

This issue is fixed by making Azul treat the content type opaquely, not appending anything to it, nor stripping anything from it.

We also need to be aware that updates (new snapshot for existing project) could modify the content-type of a file which may require updating the content-type property in info objects. This introduces the possibility of a temporary inconsistency between the mirror and the index, so I don't think a hard failure (failing assertion) is the right way to ensure consistency.

[achave11-ucsc]

Spike as described above. After that we'll create a data issue about this.

[achave11-ucsc]

Run the patched version mentioned in #7139 (comment), of the validate_snapshots.py scripts as follows to produce a JSON object with all the files matching the desired pattern, as follows:

 AZUL_DEBUG=0 python scripts/validate_snapshots.py --catalogs dcp48 --match 'dcp-type=' > offending_rows.json

Note that this takes around 19 and a half hours to check all the tables for each snapshot in dcp48.

The following is a summary of the snapshots containing the 'dcp-type=' parameter in the descriptor column in the analysis_file, image_file, sequence_file, or supplementary_file table for the snapshot.

hca_prod_005d611a14d54fbf846e571a1f874f70__20220111_dcp2_20241205_dcp45: Files with 'dcp-type=',     1768
…
hca_prod_fde199d2a8414ed1aa65b9e0af8969b1__20220330_dcp2_20230314_dcp25: Files with 'dcp-type=',      545

The following contains the total 148,482 files which contain 'dcp-type=', the table they were obtained from, the source spec, and all the additional information
offending_rows.json.zip

[edit: @hannes-ucsc, truncated long preformatted section because it is obsoleted by further analysis results posted below]