Skip to content

Backtrace when database is out of diskspace #1405

@pjonsson

Description

@pjonsson

Found this in the logs for an OWS image based on commit eda5f2b. I think this is a different issue from #1388, so putting the backtrace here for reference:

Traceback (most recent call last):
  File "/app/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/app/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 951, in do_execute
    cursor.execute(statement, parameters)
  File "/app/lib/python3.12/site-packages/psycopg/cursor.py", line 117, in execute
    raise ex.with_traceback(None)
psycopg.errors.DiskFull: could not resize shared memory segment "/PostgreSQL.2417566142" to 4206592 bytes: No space left on device
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/app/lib/python3.12/site-packages/datacube_ows/ogc.py", line 183, in ogc_svc_impl
    return version_support.router(nocase_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/datacube_ows/utils.py", line 38, in log_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/datacube_ows/wms.py", line 29, in handle_wms
    return get_map(nocase_args)
           ^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/datacube_ows/utils.py", line 38, in log_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/datacube_ows/data.py", line 107, in get_map
    n_datasets = stacker.n_datasets()
                 ^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/datacube_ows/loading.py", line 200, in n_datasets
    return self._layer.ows_index().count(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/datacube_ows/index/postgis/api.py", line 201, in count
    return layer.dc.index.datasets.count(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/datacube/index/postgis/_datasets.py", line 839, in count
    for _, count in self._do_count_by_product(query, archived=archived):
  File "/app/lib/python3.12/site-packages/datacube/index/postgis/_datasets.py", line 965, in _do_count_by_product
    count = connection.count_datasets(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/datacube/drivers/postgis/_api.py", line 831, in count_datasets
    num = self._connection.scalar(select_query)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1308, in scalar
    return meth(
           ^^^^^
  File "/app/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 544, in _execute_on_scalar
    return self._execute_on_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 526, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/app/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/app/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/app/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 951, in do_execute
    cursor.execute(statement, parameters)
  File "/app/lib/python3.12/site-packages/psycopg/cursor.py", line 117, in execute
    raise ex.with_traceback(None)
sqlalchemy.exc.OperationalError: (psycopg.errors.DiskFull) could not resize shared memory segment "/PostgreSQL.2417566142" to 4206592 bytes: No space left on device
[SQL: SELECT count(odc.dataset.id) AS count_1

OWS can't do much when the database isn't functioning, this issue is just about catching the exception in the right place so we don't get a long backtrace in the logs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions