MRG: Merge pull request #723 from octue/rc-kueue

Switch to Kueue service backend

Author: cortadocodes

.github/workflows/build-docker-image.yml

@@ -98,7 +98,7 @@ ENV/
 # See: https://github.com/google-github-actions/auth/issues/123
 google_credentials.json
 gha-creds-*.json
-gcp-creds-*.json
+gcp-cred*
 terraform/gcp-credentials.json
 .terraform
 

.gitignore

@@ -13,22 +13,24 @@ repos:
       - id: check-added-large-files
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.6.9
+    rev: v0.9.2
     hooks:
       - id: ruff
+        name: Ruff lint
         args: [--fix, --exit-non-zero-on-fix]
+
+      - id: ruff
+        name: Ruff isort
+        args: [check, --select, I, --fix]
+
       - id: ruff-format
+        name: Ruff format
 
   - repo: https://github.com/pre-commit/mirrors-prettier
     rev: v2.7.1
     hooks:
       - id: prettier
 
-  - repo: https://github.com/pycqa/pydocstyle
-    rev: 6.1.1
-    hooks:
-      - id: pydocstyle
-
   - repo: https://github.com/thclark/pre-commit-sphinx
     rev: 0.0.3
     hooks:

.pre-commit-config.yaml

@@ -63,14 +63,18 @@ You can also set the following options when you call :mod:`Child.ask <octue.reso
 - ``subscribe_to_logs`` - if true, the child will forward its logs to you
 - ``allow_local_files`` - if true, local files/datasets are allowed in any input manifest you supply
 - ``handle_monitor_message`` - if provided a function, it will be called on any monitor messages from the child
-- ``record_events`` – if ``True``, events received from the parent while it processes the question are saved to the ``Child.received_events`` property
+- ``record_events`` – if true, events received from the parent while it processes the question are saved to the ``Child.received_events`` property
 - ``save_diagnostics`` – must be one of {"SAVE_DIAGNOSTICS_OFF", "SAVE_DIAGNOSTICS_ON_CRASH", "SAVE_DIAGNOSTICS_ON"}; if turned on, allow the input values and manifest (and its datasets) to be saved by the child either all the time or just if the analysis fails
 - ``question_uuid`` - if provided, the question will use this UUID instead of a generated one
 - ``push_endpoint`` - if provided, the result and other events produced during the processing of the question will be pushed to this HTTP endpoint (a URL)
-- ``asynchronous`` - if ``True``, don't wait for an answer to the question (the result and other events can be :ref:`retrieved from the event store later <retrieving_asynchronous_answers>`)
+- ``asynchronous`` - if true, don't wait for an answer to the question (the result and other events can be :ref:`retrieved from the event store later <retrieving_asynchronous_answers>`)
+- ``cpus`` - the number of CPUs to request for the question; defaults to the number set by the child service
+- ``memory`` - the amount of memory to request for the question e.g. "256Mi" or "1Gi"; defaults to the amount set by the child service
+- ``ephemeral_storage`` - the amount of ephemeral storage to request for the question e.g. "256Mi" or "1Gi"; defaults to the amount set by the child service
 - ``timeout`` - how long in seconds to wait for an answer (``None`` by default - i.e. don't time out)
 
 If the question fails:
+
 - If ``raise_errors=False``, the unraised error is returned
 - If ``raise_errors=False`` and ``max_retries > 0``, the question is retried up to this number of times
 - If ``raise_errors=False``, ``max_retries > 0``, and ``prevent_retries_when`` is a list of exception types, the question is retried unless the error type is in the list
@@ -101,14 +105,12 @@ access the event store and run:
 
     from octue.cloud.pub_sub.bigquery import get_events
 
-    events = get_events(
-        table_id="your-project.your-dataset.your-table",
-        question_uuid="53353901-0b47-44e7-9da3-a3ed59990a71",
-    )
+    events = get_events(question_uuid="53353901-0b47-44e7-9da3-a3ed59990a71")
 
 
 **Options**
 
+- ``table_id`` - If you're not using the standard deployment, you can specify a different table here
 - ``question_uuid`` - Retrieve events from this specific question
 - ``parent_question_uuid`` - Retrieve events from questions triggered by the same parent question (this doesn't include the parent question's events)
 - ``originator_question_uuid`` - Retrieve events for the entire tree of questions triggered by an originator question (a question asked manually through ``Child.ask``; this does include the originator question's events)
@@ -219,7 +221,7 @@ This method uses multithreading, allowing all the questions to be asked at once
 
 Asking a question within a service
 ==================================
-If you have :doc:`created your own Octue service <creating_services>` and want to ask children questions, you can do
+If you have :doc:`created your own Twined service <creating_services>` and want to ask children questions, you can do
 this more easily than above. Children are accessible from the ``analysis`` object by the keys you give them in the
 :ref:`app configuration <app_configuration>` file. For example, you can ask an ``elevation`` service a question like
 this:
@@ -351,20 +353,17 @@ whole tree of children, grandchildren, and so on, please `upvote this issue.
 Using a service registry
 ========================
 When asking a question, you can optionally specify one or more `service registries
-<https://django-twined.readthedocs.io/en/latest/>`_ to resolve SRUIDs against. This is analogous to specifying a
-different ``pip`` index for resolving package names when using ``pip install``. If you don't specify any registries, the
-default Octue service registry is used.
-
-Specifying service registries can be useful if:
+<https://django-twined.readthedocs.io/en/latest/>`_ to resolve SRUIDs against. This checks if the service revision
+exists (good for catching typos in SRUIDs) and raises an error if it doesn't. Service registries can also get the
+default revision of a service if you don't provide a revision tag. Asking a question if without specifying a registry
+will bypass these checks.
 
-- You have your own private services that aren't on the default Octue service registry
-- You want services from one service registry with the same name as in another service registry to be prioritised
 
 Specifying service registries
 -----------------------------
 You can specify service registries in two ways:
 
-1. Globally for all questions asked inside a service. In the service configuration (``octue.yaml`` file):
+1. For all questions asked inside a service. In the service configuration (``octue.yaml`` file):
 
     .. code-block:: yaml
 

Dockerfile

@@ -4,42 +4,45 @@ Authentication
 You need authentication while using ``octue`` to:
 
 - Access data from Google Cloud Storage
-- Use, run, or deploy Octue services
+- Use, run, or deploy Twined services
 
-Authentication can be provided by using one of:
-
-- A service account
-- Application Default Credentials
+Authentication is provided by a GCP service account.
 
 Creating a service account
 ==========================
-1. Create a service account (see Google's `getting started guide <https://cloud.google.com/docs/authentication/getting-started>`__)
-2. Make sure your service account has access to any buckets you need, Google Pub/Sub, and Google Cloud Run if your
-   service is deployed on it (`see here <https://cloud.google.com/storage/docs/access-control/using-iam-permissions>`_)
+By setting up your Twined service network with the :doc:`Twined Terraform modules <deploying_services>`, a set of
+maintainer service accounts have already been created with the required permissions.
 
 Using a service account
 =======================
 
 Locally
 -------
-1. Create and download a key for your service account - it will be called ``your-project-XXXXX.json``.
+1. Access your service accounts `here <https://console.cloud.google.com/iam-admin/serviceaccounts>`_, making sure the
+   correct project is selected
+
+2. Click on the relevant service account, go to the "Keys" tab, and create (download) a JSON key for it - it will be
+   called ``<project-name>-XXXXX.json``.
 
 .. DANGER::
 
     It's best not to store this in your project to prevent accidentally committing it or building it into a docker
-    image layer. Instead, bind mount it into your docker image from somewhere else on your local system.
+    image layer. Instead, keep it somewhere else on your local system with any other service account keys you already
+    have.
 
-    If you must keep within your project, it's good practice to name the file ``gha-greds-<whatever>.json`` and make
-    sure that ``gha-creds-*`` is in your ``.gitignore`` and ``.dockerignore`` files.
+    If you must keep within your project, it's good practice to name the file ``gcp-credentials.json`` and make
+    sure that ``gcp-cred*`` is in your ``.gitignore`` and ``.dockerignore`` files.
 
-2. If you're developing in a container (like a VSCode ``.devcontainer``), mount the file into the container. You can
+2. If you're developing in a container (like a VSCode ``devcontainer``), mount the file into the container. You can
    make gcloud available too - check out `this tutorial
    <https://medium.com/datamindedbe/application-default-credentials-477879e31cb5>`_.
 
-3. Set the ``GOOGLE_APPLICATION_CREDENTIALS`` environment variable to the path of the key file.
+3. Set the ``GOOGLE_APPLICATION_CREDENTIALS`` environment variable to the absolute path of the key file. If using a
+   ``devcontainer``, make sure this is the path inside the container and not the path on your local machine.
 
-On GCP infrastructure
----------------------
-- Credentials are provided when running code on GCP infrastructure (e.g. Google Cloud Run)
-- ``octue`` uses these when when running on these platforms
-- You should ensure the correct service account is being used by the deployed instance
+On GCP infrastructure / deployed services
+-----------------------------------------
+- Credentials are automatically provided when running code or services on GCP infrastructure, including the Kubernetes
+  cluster
+- ``octue`` uses these when when running on these platforms, so there's no need to upload a service account key or
+  include one in service docker images

docs/source/asking_questions.rst

@@ -58,7 +58,7 @@ Accessing inputs and storing outputs
 ------------------------------------
 Your app must access configuration and input data from and store output data on the :mod:`analysis <octue.resources.analysis.Analysis>`
 parameter (for function-based apps) or attribute (for class-based apps). This allows standardised
-configuration/input/output validation against the twine and interoperability of all Octue services while leaving you
+configuration/input/output validation against the twine and interoperability of all Twined services while leaving you
 freedom to do any kind of computation. To access the data, use the following attributes on the ``analysis``
 parameter/attribute: