API in production is unstable and returns 500 errors

[tloubrieu-jpl]

Checked for duplicates

No - I haven't checked

🐛 Describe the bug

Reported by Dan Scholes:
https://pds.mcp.nasa.gov/api/search/1/products/urn:nasa:pds:izenberg_pdart14_meap:data_grsspectra
Error:
500 INTERNAL_SERVER_ERROR Request uri=/api/search/1/products/urn:nasa:pds:izenberg_pdart14_meap:data_grsspectra failed with message: Unhandled Exception: Read timed out(ref:c268b436-5a1f-4520-b831-63edc5c572ab) For assistance, forward this error message to pds-operator@jpl.nasa.gov

Worked:
https://pds.mcp.nasa.gov/api/search/1/products/urn:nasa:pds:izenberg_pdart14_meap:data_grsspectra:grs_rac2004317zzz

Error:
https://pds.mcp.nasa.gov/api/search/1/products/urn:nasa:pds:izenberg_pdart14_meap:data_grsspectra/members

Bundle worked:
https://pds.mcp.nasa.gov/api/search/1/products/urn:nasa:pds:mro_crism_ddr_1001

Collection failed:
https://pds.mcp.nasa.gov/api/search/1/products/urn:nasa:pds:mro_crism_ddr_1001:data_ddr

Product worked (but didn’t earlier)
https://pds.mcp.nasa.gov/api/search/1/products/urn:nasa:pds:mro_crism_ddr_1001:data_ddr:hrl0000bf18_0d_de156l_ddr1

Reported by Sean Kelly:
HTTP response body: 500 INTERNAL_SERVER_ERROR
Request uri=/api/search/1/products failed with message:
Unhandled Exception: Read timed out(ref:c0e782d7-d0d5-4fbf-b11c-4acc7367b19f)
For assistance, forward this error message to pds-operator@jpl.nasa.gov

Reported by Anil Natha
This afternoon I've been experiencing inconsistent responses with the API. Sometimes it's fast, working as normal. Then if I repeat the same call just seconds later, it's slow, but does eventually respond. And a couple times I received a 500 error:
500 INTERNAL_SERVER_ERROR Request uri=/api/search/1/products failed with message: Unhandled Exception: Read timed out(ref:d91101ff-d280-4992-a13f-0240cd6d5902) For assistance, forward this error message to pds-operator@jpl.nasa.gov
The API call I made is: https://pds.mcp.nasa.gov/api/search/1/products?q=(lid%20eq%20%22urn:nasa:pds:ast-sat.thomas.shape-models:data%22). Just wanted to report the erratic behavior.

The planetPatrol tool is also reported the intabilities.

🕵️ Expected behavior

I expected to get the results without error and in a timely manner (less than 1s)

📜 To Reproduce

Transient issue to be investigated

🖥 Environment Info

In production

📚 Version of Software Used

No response

🩺 Test Data / Additional context

No response

🦄 Related requirements

🦄 #xyz

⚙️ Engineering Details

No response

🎉 Integration & Test

No response

[tloubrieu-jpl]

Hi @scholes-ds, we are tracking your reported issue here.