Enhance monitoring lambda to catch Panel apps failing with 500

[maximlt]

The volume imaging app (https://volumetric-imaging.holoviz-demo.anaconda.com) is currently broken, visiting the app returns 500: Internal Server Error. The lambda that checks hourly the status of the deployments (reporting to Discord) didn't catch this.

The lambda's code (https://gist.github.com/maximlt/f2e29eb7e6dbcaa4da4ee7d68636ac55, not automatically synced with AWS) only makes a HEAD query to the root app endpoint (https://volumetric-imaging.holoviz-demo.anaconda.com) and checks if the status code is 405.

Making instead a GET query returns a 302 (found redirect). Following the redirect to https://volumetric-imaging.holoviz-demo.anaconda.com/volumetric_imaging where the Panel app is actually served, a GET query returns a 500.

In order to better check all the Panel deployments, we'd need to switch to a GET query. In fact, the best would be to run some UI/Playwright script against each app to make sure they really work. But let's keep it simple for now. Making a GET query to all the apps simultaneously might overload the server? A better solution may be to re-deploy the Panel apps with --liveness enabled and hit /liveness. Though I'm not entirely sure hitting it would catch this 500. I've seen /liveness also accepts an endpoint argument to test a specific deployed app (e.g. /liveness?endpoint=/volumetric_imaging). The liveness handler then runs that (still not sure that'd be enough to catch the 500 without testing for real):

        app = self.applications[endpoint]
        try:
            doc = app.create_document()
            _cleanup_doc(doc)
            self.write({endpoint: True})
        except Exception as e:
            raise web.HTTPError(
                500, f"Endpoint {endpoint!r} could not be served. Application raised error: {e}"
            ) from e