 -[**Projects**](/docs/users/projects/projects/) in Renku may contain [**Data**](/docs/users/data/data/), [**Code**](/docs/users/code/code-repository) and [**Sessions**](/docs/users/sessions/environment/) to foster [**Collaboration**](/docs/users/collaboration/). Check each individual building block for understanding the possibilities that Renku offers. +[**Projects**](/docs/users/projects/projects/) in Renku may contain [**Data**](/docs/users/data/data/), [**Code**](/docs/users/code/code-repository) and [**Sessions**](/docs/users/compute/session/) to foster [**Collaboration**](/docs/users/collaboration/). Check each individual building block for understanding the possibilities that Renku offers.
--- diff --git a/docs/docs/10-users/20-getting-started/50-launch-session.md b/docs/docs/10-users/20-getting-started/50-launch-session.md index 7f650ae276..598e2275ed 100644 --- a/docs/docs/10-users/20-getting-started/50-launch-session.md +++ b/docs/docs/10-users/20-getting-started/50-launch-session.md @@ -4,15 +4,16 @@ To run your code and analyze data on RenkuLab, first decide what kind of session environment you would like by creating a session launcher. -1. In the **Sessions** section of your project page, click on **+**. -2. In the **Global environment** tab, scroll down and select **Python Data Science - Jupyter**. This will give you a pre-configured environment with python and data science packages pre-installed. If you need to install other packages, you can install any necessary packages using the terminal. For a detailed guide for how to install packages, see [How to install packages on-the-fly in your session](../sessions/guides/environments/install-packages-on-the-fly-in-your-session). However, the packages you install on-the-fly in the session will not be shared with other people working with your Renku project. To ensure the adequate dependencies are installed in your environment, follow step 4.3 to create a custom environment from code. -3. Click **Next**. -4. Choose the compute resources for your session based on your project’s needs in the drop-down menu, and select the amount of disk space. +1. In the **Launchers** section of your project page, click on **+**. +2. Select **Session Launcher** card. +3. In the **Global environment** tab, scroll down and select **Python Data Science - Jupyter**. This will give you a pre-configured environment with python and data science packages pre-installed. If you need to install other packages, you can install any necessary packages using the terminal. For a detailed guide for how to install packages, see [How to install packages on-the-fly in your session](../compute/session/guides/install-packages-on-the-fly-in-your-session). However, the packages you install on-the-fly in the session will not be shared with other people working with your Renku project. To ensure the adequate dependencies are installed in your environment, follow step 4.3 to create a custom environment from code. +4. Click **Next**. +5. Choose the compute resources for your session based on your project’s needs in the drop-down menu, and select the amount of disk space.  :::info -You can have multiple Session Launchers in your project that run different kinds of sessions. Check out [How to guides to add a session launcher to your project](../sessions/guides/environments) +You can have multiple Session Launchers in your project that run different kinds of sessions. Check out [How to guides to add a session launcher to your project](../compute/session/guides/add-a-session-launcher-to-your-project) ::: :::info @@ -29,7 +30,7 @@ Once your session has launched, you will see the data and code that you linked t ## Add a second Session Launcher (optional) -When you want to create a RenkuLab session with a set of packages that’s reproducible and shareable, we recommend defining your package dependencies in code. Then, use the ‘Create from Code’ session launcher option to have Renku create a reproducible session environment for you! See [How to create an environment with custom packages installed](../sessions/guides/environments/create-environment-with-custom-packages-installed) for details. +When you want to create a RenkuLab session with a set of packages that’s reproducible and shareable, we recommend defining your package dependencies in code. Then, use the ‘Create from Code’ session launcher option to have Renku create a reproducible session environment for you! See [How to create an environment with custom packages installed](../compute/environment/guides/create-environment-with-custom-packages-installed) for details. ## Run the Code (optional) diff --git a/docs/docs/10-users/30-projects/00-projects.md b/docs/docs/10-users/30-projects/00-projects.md index a8ca1ec3d5..ac60075288 100755 --- a/docs/docs/10-users/30-projects/00-projects.md +++ b/docs/docs/10-users/30-projects/00-projects.md @@ -4,9 +4,9 @@ A Renku project can contain: - [Code repository](../code/code-repository) - [Data connector](../data/data) -- [Session Launcher](../sessions/session), which brings together: - - [Environment](../sessions/environment) - - [Resource Pools & Classes](../sessions/resource-pools-and-classes) +- [Session Launcher](../compute/session), which brings together: + - [Environment](../compute/environment) + - [Resource Pools & Classes](../compute/resource-pools-and-classes) A Renku project may have members; see [Permissions, Roles, and Access Rights](/docs/users/collaboration/permissions). diff --git a/docs/docs/10-users/30-projects/guides/10-create-a-project.md b/docs/docs/10-users/30-projects/guides/10-create-a-project.md index 79149d93c9..5619937ace 100755 --- a/docs/docs/10-users/30-projects/guides/10-create-a-project.md +++ b/docs/docs/10-users/30-projects/guides/10-create-a-project.md @@ -25,9 +25,9 @@ To create a new project: 4. Add code repository as explained in [How to add a code repository to your project](../../code/guides/add-code-repository-to-project). 5. Create a session launcher for working in your project: 1. Select an environment: - 1. Check out the environments available in Renku via [How to add a session launcher to your project](../../sessions/guides/environments/). If these are sufficient for your project, use them! - 2. If you need to customize the environment for your project, you can create a custom environment yourself via [How to use your own docker image for a Renku session](../../sessions/guides/environments/use-your-own-docker-image-for-renku-session), or [Contact](../../community) us if you are in trouble! - 2. Set the session launcher’s default resource class for your project [How to select compute resources for your session](../../sessions/guides/select-compute-resources-for-session). + 1. Check out the environments available in Renku via [How to add a session launcher to your project](../../compute/session/guides/add-a-session-launcher-to-your-project). If these are sufficient for your project, use them! + 2. If you need to customize the environment for your project, you can create a custom environment yourself via [How to use your own docker image for a Renku session](../../compute/environment/guides/use-your-own-docker-image-for-renku-session), or [Contact](../../community) us if you are in trouble! + 2. Set the session launcher’s default resource class for your project [How to select compute resources for your session](../../compute/guides/select-compute-resources-for-session). :::info diff --git a/docs/docs/10-users/50-code/00-code-repository.md b/docs/docs/10-users/50-code/00-code-repository.md index 2b16af734a..6c3f2c1129 100755 --- a/docs/docs/10-users/50-code/00-code-repository.md +++ b/docs/docs/10-users/50-code/00-code-repository.md @@ -7,7 +7,7 @@ For the purposes of Renku, a code repository is a git-based collection of code f When a code repository is connected to a [Project](../projects/projects), the repository is automatically cloned into the -[Session](../sessions/session), where the files can be accessed and run. +[Session](../compute/session), where the files can be accessed and run. ## Access to Code Repositories from RenkuLab diff --git a/docs/docs/10-users/60-compute/10-launcher.md b/docs/docs/10-users/60-compute/10-launcher.md new file mode 100755 index 0000000000..165ce5461e --- /dev/null +++ b/docs/docs/10-users/60-compute/10-launcher.md @@ -0,0 +1,34 @@ +# Launcher + +A launcher is a component of a project that defines how compute runs on RenkuLab. Each launcher combines an [Environment](environment) with a default [Resource Pools & Classes](resource-pools-and-classes) configuration. + +Renku supports two types of launchers: + +1. **Session launcher:** launches an interactive, browser-based [Session](session) for coding and data exploration. +2. **Job launcher:** runs a non-interactive [Job](job) in the background when you submit it. + ++ +
+ +A launcher contains: + +1. an [Environment](environment), which defines the software packages and libraries used by the session or job, and +2. a default resource class (see [Resource Pools & Classes](resource-pools-and-classes)), which determines the availabler compute resources. + +For a Job launcher, you also configure a **job command** that defines what runs when the job is submitted. + +Project's launchers are usable to everyone who can see the project. Anyone with access to the project can launch a session or submit a job from a launcher. + +However, the sessions you launch and the jobs you submit are only accessible to you. They are not shared across users. + +:::info + +**App Launchers** will be available in October 2026. + +::: + +## Next steps + +- To create an interactive session launcher, see [Add a session launcher to your project](session/guides/add-a-session-launcher-to-your-project). +- To create a job launcher, see [Add a job launcher to your project](job/guides/create-a-job-launcher). diff --git a/docs/docs/10-users/60-compute/20-resource-pools-and-classes.md b/docs/docs/10-users/60-compute/20-resource-pools-and-classes.md new file mode 100755 index 0000000000..cf6029012d --- /dev/null +++ b/docs/docs/10-users/60-compute/20-resource-pools-and-classes.md @@ -0,0 +1,38 @@ +# Resource Pools & Classes + +Compute resources available to Renku users are determined by the **resources pools** the user has access to. To specify the resources for a project’s launcher, you select a **resource class** from any of the resource pools available to you. The same resource pools and classes apply to interactive [Sessions](session) and background [Jobs](job). + +## Resource Pools + +A **resource pool** consists of a total resource quota assigned to a set of users who are allowed to use resources from that quota with culling times for sessions. + +- **Resource quota:** For example, a resource pool may be configured for a total of 160 CPUs, 1024 GB RAM, and 16 GPUs. +- **Users:** There is no limit to how many users can be added to a resource pool. And conversely, a user may be granted access to any number of resource pools. All resource pools the user has access to show up when defining or modifying the default resource class for a launcher. +- **Culling times:** The maximum idle time for a resource pool(s) can be configured by the administrator. For example, in the Public resource pool (the default resource pool available to all RenkuLab users), interactive sessions auto-pause after 2 hours of idle time, and shut down after 2 weeks. These thresholds may be customized to be longer or shorter. Jobs are not paused for idle time in the same way as sessions, instead completed and errored jobs are removed from the interface after a retention period (see [Job](job)). + +## Resource Classes + +A resource pool contains a set of **resource classes**. A resource class is a resource configuration that a user can select for their RenkuLab session or job. + +For example, a resource class may be: + +- 12 CPUs, 80 GB RAM, 2 GPUs, 50 GB _default_ disk, 500 GB _max_ disk + +As users run sessions and jobs, their compute resources (the resource class in use) come out of the quota on the resource pool. If the resource pool’s quota has been reached, users won’t be able to start more sessions or jobs in that resource pool (but they may use other resource pools they have access to). + ++ +*For custom resource pools granted to users, you may customize the resource classes in your resource pool.* +
+ +## The Public Resource Pool & Classes + +On RenkuLab, the following resources classes are publicly available (in the “public resource pool”): + ++ +
+ +## Request a Custom Resource Pool {#request-custom-resource-pool} + +Do you need more resources than are available in RenkuLab’s public resource classes? [Contact](../community) us! We can configure a custom resource pool for your needs upon demand. We apply flat, transparent pricing, passing down infra rates. diff --git a/docs/docs/10-users/60-compute/40-environment/_category_.yml b/docs/docs/10-users/60-compute/40-environment/_category_.yml new file mode 100644 index 0000000000..2333c4f68b --- /dev/null +++ b/docs/docs/10-users/60-compute/40-environment/_category_.yml @@ -0,0 +1 @@ +label: Environment diff --git a/docs/docs/10-users/60-sessions/guides/00-environments/20-create-environment-with-custom-packages-installed.md b/docs/docs/10-users/60-compute/40-environment/guides/10-create-environment-with-custom-packages-installed.md similarity index 97% rename from docs/docs/10-users/60-sessions/guides/00-environments/20-create-environment-with-custom-packages-installed.md rename to docs/docs/10-users/60-compute/40-environment/guides/10-create-environment-with-custom-packages-installed.md index 37c3165e0b..0e35e003c7 100755 --- a/docs/docs/10-users/60-sessions/guides/00-environments/20-create-environment-with-custom-packages-installed.md +++ b/docs/docs/10-users/60-compute/40-environment/guides/10-create-environment-with-custom-packages-installed.md @@ -230,7 +230,7 @@ repository](create-environment-with-custom-packages-private-code-repository). 1. Make sure the code repository that contains your environment definition file is added to your Renku project. -2. Create a **new session launcher** +2. Create a **new session launcher** by clicking the "+" button in the Launchers section, and select either Session Launcher or Job Launcher. 3. Select the **Create from code** option  @@ -258,7 +258,7 @@ repository](create-environment-with-custom-packages-private-code-repository). 7. Click **Next** 8. Define the **name** of the Session Launcher 9. Select the default **compute resources** -10. Click on **Add session launcher** +10. Click on **Add launcher** The environment is now being built by RenkuLab. You can see the status on the session launcher. @@ -272,8 +272,8 @@ When the environment is built, you can launch your session. update the environment definition file in the code repository where the environment is defined. 2. Then, rebuild the environment in RenkuLab: - 1. Click on the session launcher to open the session launcher side panel. - 2. Navigate to the **Session Environment** section. + 1. Click on the launcher to open the launcher side panel. + 2. Navigate to the **Environment** section. 3. Click on **Rebuild**. ## [experimental] Using a dashboard with a code-based environment diff --git a/docs/docs/10-users/60-sessions/guides/00-environments/25-create-environment-with-custom-packages-private-code-repository.md b/docs/docs/10-users/60-compute/40-environment/guides/20-create-environment-with-custom-packages-private-code-repository.md similarity index 100% rename from docs/docs/10-users/60-sessions/guides/00-environments/25-create-environment-with-custom-packages-private-code-repository.md rename to docs/docs/10-users/60-compute/40-environment/guides/20-create-environment-with-custom-packages-private-code-repository.md diff --git a/docs/docs/10-users/60-compute/40-environment/guides/30-use-your-own-docker-image-for-renku-session.md b/docs/docs/10-users/60-compute/40-environment/guides/30-use-your-own-docker-image-for-renku-session.md new file mode 100755 index 0000000000..991175c6ff --- /dev/null +++ b/docs/docs/10-users/60-compute/40-environment/guides/30-use-your-own-docker-image-for-renku-session.md @@ -0,0 +1,106 @@ +# Use your own docker image for a launcher + +## What Docker images can I use in RenkuLab sessions or jobs? + +There are some limitations to what images you can use in Renku sessions. The image must meet **all** of the conditions below in order to work on RenkuLab: + +1. **The image needs a front end (only applies to session launchers)** + + The image you use needs a user interface web frontend to run so that you can access the session via the browser. Examples of these include JupyterLab, VSCode or RStudio. + +2. **The image will be run as non-root** + + For security reasons, sessions are started with a non-root user. + +## How to configure an image as a custom environment + +In the project page: + +1. Under **Launchers** section click on ➕ to add a new session or job launcher +2. Choose **Session** or **Job** in the type selector. +3. Select **External environment** + ++ +
+ +3. For the container image, provide an **image identifier**. + - Some examples of image identifiers: + - if the image is hosted on DockerHub: + - `renku/renkulab-py:3.10-0.24.0` + - `continuumio/anaconda3:2024.06-1` + - if the image is hosted on gitlab.renkulab.io: + - `registry.renkulab.io/laura.kinkead1/n2o-pathway-analysis:980f4a3` + - The image identifier should be in the format that works with `docker pull` +4. Depending on the image you’re using, you’ll need to fill in the **Advanced settings**. See [Example image configurations for common front ends](./configure-frontends-for-a-session-environment) for ready-to-use configurations, or the information below for how to fill it in: + + :::danger + + This part is important! Please read carefully. + + ::: + - I’m using an image created by **Renku** and that is **newer** than version 0.24.0 (the version number is in the image tag). + + The only additional parameter you have to provide in the session launcher creation dialog is the `Default URL` and this should be set to `/lab`. + +  + + - I’m using an image created by **Renku** and that is **older** than version 0.24.0 (the version number is in the image tag). + + :::note + + If you are working with an image in a launcher where the **launcher was created before November 27, 2024**, the launcher was migrated automatically with the new Renku release to include the necessary advanced settings. The instructions below apply only to new session launchers you are creating for the first time. + + ::: + + For Renku base images of version 0.24.0 or older (or images that are based on these images), you have 2 options: + 1. **Option 1:** Upgrade your base image to 0.25.0 or newer. This can be done by going into the settings of the Renku 1.0 project that builds the image, and accepting the updates. Or, directly update your Dockerfile to refer to the newer base image. + 2. **Option 2:** Provide additional configuration in the launcher. Here is an example configuration needed to run a Renku base image of version 0.24.0 or older: + - **Container Image**: `renku/renkulab-py:3.10-0.24.0` or whatever image you are trying to use + - **Default URL**: `/lab` (or `/rstudio` if you are using `renku/renkulab-r` or `renku/renkulab-bioc`). (only applies to session launchers) + - **Mount Directory**: `/home/jovyan/work` + - **Working Directory**: `/home/jovyan/work` + - **UID**: `1000` + - **GID**: `100` + - **Command ENTRYPOINT**: `["sh", "-c"]` + - **Command Arguments**: + + ```json + [ + "/entrypoint.sh jupyter server --ServerApp.ip=0.0.0.0 --ServerApp.port=8888 --ServerApp.base_url=$RENKU_BASE_URL_PATH --ServerApp.token=\"\" --ServerApp.password=\"\" --ServerApp.allow_remote_access=true --ContentsManager.allow_hidden=true --ServerApp.allow_origin=* --ServerApp.root_dir=\"/home/jovyan/work\"" + ] + ``` + + - I’m using an image created **somewhere else** (not by Renku). + + You need to fill in the **Advanced Settings** for your image to work on RenkuLab. See [Example image configurations for common front ends](./configure-frontends-for-a-session-environment). + +5. Select the **Resource class** that best fits your expected computational needs. + + :::tip + + If the available resource classes are too small for your compute requirements, we can create a custom resource pool for you! See [Request a Custom Resource Pool](../../resource-pools-and-classes#request-custom-resource-pool). + + ::: + +6. Give your session or job launcher a **name** +7. Click on **Add launcher** button + +:::info + +Note that you can always **modify your launcher** by clicking on top of it on the project’s page, and using the menu on the right: + ++ +
+ +::: + +## About Renku Session URLs {#about-renku-session-urls} + +The biggest challenge with running custom images on Renku is managing the URL path where the session is accessible. This path is not known ahead of time but only once the session has been launched. Renku injects two environment variables in each session to indicate the full session URL and the path portion of the URL. These environment variables are named respectively `RENKU_BASE_URL` and `RENKU_BASE_URL_PATH`. Regardless of what image you are running on Renku you will have to specify the path where the session can be accessed. Most programs you will run in an image will assume that the path where they run is `/`, but we never run session at such location. For example, on [renkulab.io](http://renkulab.io) sessions are available at URLs like the following: `https://renkulab.io/sessions/tasko-olevsk-bfff446a2f41` + +For the session available at the URL above, the environment variables have the following values: + +- `RENKU_BASE_URL` = `https://renkulab.io/sessions/tasko-olevsk-bfff446a2f41` +- `RENKU_BASE_URL_PATH` = `/sessions/tasko-olevsk-bfff446a2f41` diff --git a/docs/docs/10-users/60-compute/40-environment/guides/40-configure-frontends-for-a-session-environment.md b/docs/docs/10-users/60-compute/40-environment/guides/40-configure-frontends-for-a-session-environment.md new file mode 100644 index 0000000000..052996e1c7 --- /dev/null +++ b/docs/docs/10-users/60-compute/40-environment/guides/40-configure-frontends-for-a-session-environment.md @@ -0,0 +1,160 @@ +# Configure front ends for a session environment + +In order to run a docker image in a session, Renku needs to know some information about how to run and serve that image. + +In this section, you can see example configurations for commonly used images. If you build an image with one of these images as the base, then you can use this provided configuration to make that image run in RenkuLab. The information below can be copied and pasted into the **Advanced Settings** form for creating a **custom environment**. + +This page provides reference configurations to use in the **Advanced Settings** step of [How to use your own docker image for a Renku session](./use-your-own-docker-image-for-renku-session), specifically for images not built by Renku. + ++ +
+ +### Jupyter + +- Container Image: `jupyter/minimal-notebook:python-3.11` +- Port: `8888` +- Default URL: `/lab` +- Command ENTRYPOINT: + +```json +["sh", "-c"] +``` + +- Command Arguments CMD ([learn more](./use-your-own-docker-image-for-renku-session#about-renku-session-urls)): + +```json +[ + "jupyter server --ServerApp.ip=0.0.0.0 --ServerApp.port=8888 --ServerApp.base_url=$RENKU_BASE_URL_PATH --ServerApp.token=\"\" --ServerApp.password=\"\" --ServerApp.allow_remote_access=true --ContentsManager.allow_hidden=true --ServerApp.allow_origin=*" +] +``` + +### Julia and Jupyter + +- Container Image: `jupyter/julia-notebook:x86_64-python-3.11.6` +- Port: `8888` +- Default URL: `/lab` +- Working Directory: `/home/jovyan/work` +- Mount Directory: `/home/jovyan/work` +- Command ENTRYPOINT: + +```json +["sh", "-c"] +``` + +- Command Arguments CMD ([learn more](./use-your-own-docker-image-for-renku-session#about-renku-session-urls)): + +```json +[ + "jupyter server --ServerApp.ip=0.0.0.0 --ServerApp.port=8888 --ServerApp.base_url=$RENKU_BASE_URL_PATH --ServerApp.token=\"\" --ServerApp.password=\"\" --ServerApp.allow_remote_access=true --ContentsManager.allow_hidden=true --ServerApp.allow_origin=*" +] +``` + +### RStudio + +🚧 _Not yet available_ + +### RShiny + +🚧 _Not yet available_ + +### VSCode + +- Container Image: Build a docker image that includes vscode in the PATH +- Port: `8888` +- Command ENTRYPOINT: + +```json +["sh", "-c"] +``` + +- Command Arguments CMD ([learn more](./use-your-own-docker-image-for-renku-session#about-renku-session-urls)): + +```json +[ + "code serve-web --server-base-path $RENKU_BASE_URL_PATH/ --without-connection-token --host 0.0.0.0 --port 8888" +] +``` + +### Streamlit + +- Container Image: Build a docker image that includes streamlit and any other requirements needed by your streamlit app +- Port: `8888` +- Command ENTRYPOINT: + +```json +["sh", "-c"] +``` + +- Command Arguments CMD (fill in `-  + 
::: @@ -74,7 +74,7 @@ Fill in the form so that a RenkuLab admin can let you launch sessions with the C Note: If you see the following screen, wait a minute and try to refresh the page. The session is still starting at CSCS.-  + 
::: @@ -101,7 +101,7 @@ Fill in the form so that a RenkuLab admin can let you launch sessions with the C **Currently working** :white_check_mark: Launch interactive Renku sessions at CSCS on: `Eiger` and `Bristen` - - Sessions on `Daint` , `Clariden` work, but only with special images created by the Renku team. Simplified integration with ARM clusters is coming soon. Please [Contact us](/docs/users/community) for more info. + - Sessions on `Daint` , `Clariden` work, but only with special images created by the Renku team. Simplified integration with ARM clusters is coming soon. Please [Contact us](../../../community) for more info. :white_check_mark: Access data from scratch, store and home @@ -116,7 +116,7 @@ Fill in the form so that a RenkuLab admin can let you launch sessions with the C [See the example Project "Demo HPC"](https://staging.dev.renku.ch/p/flora.thiebaut/demo-hpc#launcher-01K58XQSDV5HFF0T9A6D1G16ZF) -  +  :white_check_mark: “Create from code” environments and Global environments work (on Eiger only) diff --git a/docs/docs/10-users/60-sessions/guides/00-environments/30-install-packages-on-the-fly-in-your-session.md b/docs/docs/10-users/60-compute/50-session/guides/30-install-packages-on-the-fly-in-your-session.md similarity index 69% rename from docs/docs/10-users/60-sessions/guides/00-environments/30-install-packages-on-the-fly-in-your-session.md rename to docs/docs/10-users/60-compute/50-session/guides/30-install-packages-on-the-fly-in-your-session.md index a93b32d1ff..8033c3920a 100755 --- a/docs/docs/10-users/60-sessions/guides/00-environments/30-install-packages-on-the-fly-in-your-session.md +++ b/docs/docs/10-users/60-compute/50-session/guides/30-install-packages-on-the-fly-in-your-session.md @@ -1,12 +1,27 @@ # Install packages on-the-fly in your session +You can install packages directly in a running session's terminal to try them out right away, without editing project files or rebuilding your environment. + +:::note + +Packages installed this way: + +- Are only available to you and they are lost when you stop and restart the session, so you'll need to reinstall them every time you launch or resume the session. +- Are not shared with anyone else who launches a session from the same launcher. + +If you'd like a package to persist and be available to everyone who uses the launcher, add it to your project's environment definition file, rebuild the launcher and restart the session. See [How to create an environment with custom packages installed](../../environment/guides/create-environment-with-custom-packages-installed). + +::: + **Choose your language:** ## Python -Renku can build an environment based on a Python dependencies file for you (no knowledge of Docker required)! +Open a terminal in your session and run: -To get started, see [How to create an environment with custom packages installed](create-environment-with-custom-packages-installed). +```bash +pip install+ +
+ +2. In the **Select the type of launcher to create** modal, choose **Job**. + ++ +
+ +3. Choose and configure an **environment**: + - **Create from code** — Renku builds an environment from a code repository (for example, one with an `environment.yml` or `requirements.txt`). See [How to create an environment with custom packages installed](../../environment/guides/create-environment-with-custom-packages-installed). + - **External environment** — use your own Docker image. See [How to use your own docker image for a Renku session](../../environment/guides/use-your-own-docker-image-for-renku-session). + ++ +
+ +4. Configure the **job command** as a JSON array, for example `["python", "my_repo/main.py"]`. This command is required and defines what runs when someone submits a job from this launcher. + + You can optionally set **job args** as a separate JSON array. + +5. Set a **name** and optional **description** for the launcher. + +6. Select the **resource class** that best fits your expected computational needs. (see [Resource Pool and Classes](../../resource-pools-and-classes)) + + :::tip + + Do you need more resources than are available in RenkuLab's public resource classes? [Contact us](../../../community)! We can configure a custom resource pool for your needs upon demand. + + ::: + +7. Click **Add launcher** (or the equivalent button to save the launcher). + ++ +
+ +If your environment is built from code, wait for the environment **build** to succeed before submitting a job. You can rebuild the environment from the launcher card if needed. + +## Next steps + +Once your job launcher is ready, see [How to submit a job](submit-a-job). diff --git a/docs/docs/10-users/60-compute/60-job/guides/20-submit-a-job.md b/docs/docs/10-users/60-compute/60-job/guides/20-submit-a-job.md new file mode 100644 index 0000000000..838064b2ed --- /dev/null +++ b/docs/docs/10-users/60-compute/60-job/guides/20-submit-a-job.md @@ -0,0 +1,54 @@ +# Submit a job + +When a job launcher is configured and its environment image is ready, anyone who can see the project can **Submit** a job from that launcher. Each submission runs the job command in the background. + +## Before you submit + +Renku may ask you to resolve prerequisites before the job starts, which are the same kinds of checks that apply when launching a [Session](../../session): + +- **Code repositories** that Renku cannot access. +- **Session secrets** that you have not provided for the project yet. +- **Data connector** credentials. + +See [How to configure a session secret](../../guides/configure-session-secret) for setting up secrets that jobs can use. + +## Submit a job + +1. On the project page, find the job launcher and click **Submit**. + ++ +
+ +2. Review the environment summary in the **Review and submit job** modal. + ++ +
+ +3. Enter a **Submission ID**, that is a short unique name for this job run (for example, `run-a1b2c3`). Renku suggests one automatically but you can edit it before submitting the job. + + The submission ID must: + - Start with a lowercase letter + - Use only lowercase letters, numbers, and hyphens + - Be at least 4 characters long + + If a job with the same submission ID already exists on this launcher, Renku warns you in the form. You cannot submit until you choose a different submission ID. + +4. Confirm the **Job command** (required). It is pre-filled from the launcher configuration. You can override it for this submission if needed. The command must be a JSON array, for example `["python", "my_repo/main.py"]`. + +5. Optionally set **Job args** as a JSON array. + +6. Select a **resource class** for this submission, if you want to override the launcher defaults. + +7. Click **Submit job**. + ++ +
+ +You can submit multiple jobs from the same launcher at the same time. Each job must have its own submission ID. + +## After you submit + +While the job is starting or running, see [How to monitor, cancel, and dismiss jobs](monitor-cancel-and-dismiss-jobs). diff --git a/docs/docs/10-users/60-compute/60-job/guides/30-monitor-cancel-and-dismiss-jobs.md b/docs/docs/10-users/60-compute/60-job/guides/30-monitor-cancel-and-dismiss-jobs.md new file mode 100644 index 0000000000..854b6b900b --- /dev/null +++ b/docs/docs/10-users/60-compute/60-job/guides/30-monitor-cancel-and-dismiss-jobs.md @@ -0,0 +1,70 @@ +# Monitor, cancel, and dismiss jobs + +After you submit a [Job](../../job), you can follow its progress on the project page, in the launcher side panel, and on your Renku dashboard. + +## Where to find your jobs + +Your active and recent jobs appear in several places: + +- **Under the job launcher** on the project page — each row shows the submission ID and status +- **In the launcher side panel** — open the launcher to see **Your submitted jobs** +- **On your dashboard** — jobs are listed just below interactive sessions + ++ +
+ +## Job states + +| State | What it means | +| ------------- | --------------------------------------- | +| **Starting** | Renku is preparing the job | +| **Running** | The job command is executing | +| **Completed** | The job finished successfully | +| **Errored** | The job failed | +| **Stopping** | Renku is cancelling or removing the job | + +## View logs + +While a job is **Starting**, **Running**, **Completed**, or **Errored**, click **View logs** to open the job logs. + ++ +
+ +Logs are available for 24 hours or until you dismiss the job. + +## Cancel a running job + +If a job is still **Starting** or **Running**, you can cancel it: + +1. Click **Cancel** on the job row or in the launcher panel. +2. Confirm in the modal. Cancelling stops the job and removes it. + ++ +
+ +## Dismiss a completed or errored job + +When a job has **Completed** or **Errored**, click **Dismiss** to remove it from your project and dashboard. + ++ +
+ +:::warning + +Dismissing a job is permanent. You will not be able to view that job or its logs again in RenkuLab after dismissing it. + +::: + ++ +
+ +## Multiple jobs at once + +You can have multiple jobs running from the same launcher at the same time. Use a different **submission ID** for each job so you can tell them apart. + +Resource limits still apply at the resource pool level — if a pool's quota is full, new jobs (and sessions) may not start until resources are available. See [Resource Pools & Classes](../../resource-pools-and-classes). diff --git a/docs/docs/10-users/60-sessions/guides/20-handle-long-training-runs.md b/docs/docs/10-users/60-compute/60-job/guides/40-handle-long-training-runs.md similarity index 72% rename from docs/docs/10-users/60-sessions/guides/20-handle-long-training-runs.md rename to docs/docs/10-users/60-compute/60-job/guides/40-handle-long-training-runs.md index 37083ac4e1..750fe619ea 100644 --- a/docs/docs/10-users/60-sessions/guides/20-handle-long-training-runs.md +++ b/docs/docs/10-users/60-compute/60-job/guides/40-handle-long-training-runs.md @@ -1,6 +1,10 @@ # Handle long training runs -Long-running activities, such as model training runs, can benefit from a special workflow in your session. Sessions will stay running as long as there is user activity in the window or the CPU is used, but some cases the front-end application (e.g., JupyterLab or VSCode) may stop long-running processes. +Long-running activities, such as model training runs, are often better suited to a Renku [Job](../../job) than to an interactive session. Jobs run your command in the background without relying on a browser window or notebook kernel staying open. See [How to submit a job](submit-a-job) to get started. + +The rest of this guide describes a workflow for long-running work **inside an interactive session** when a Job is not the right fit, or when you need to keep using a session you already have open. + +Long-running activities in a session can benefit from a special workflow. Sessions will stay running as long as there is user activity in the window or the CPU is used, but in some cases the front-end application (e.g., JupyterLab or VSCode) may stop long-running processes. In this situation, you should use `tmux`, a terminal multiplexer, to keep your session running. @@ -53,7 +57,7 @@ tmux python training.py ``` -N.b., if your session launcher environment was built from code on RenkuLab, `tmux` will be available. If the environment is built elsewhere, you may need to install `tmux` yourself. +If your session launcher environment was built from code on RenkuLab, `tmux` will be available. If the environment is built elsewhere, you may need to install `tmux` yourself. ## Reconnect diff --git a/docs/docs/10-users/60-compute/60-job/guides/_category_.yml b/docs/docs/10-users/60-compute/60-job/guides/_category_.yml new file mode 100644 index 0000000000..8cca0baa51 --- /dev/null +++ b/docs/docs/10-users/60-compute/60-job/guides/_category_.yml @@ -0,0 +1,2 @@ +label: How to... +collapsed: false diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-creation-10.png b/docs/docs/10-users/60-compute/60-job/guides/job-creation-10.png new file mode 100644 index 0000000000..cb718289a4 Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-creation-10.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-creation-20.png b/docs/docs/10-users/60-compute/60-job/guides/job-creation-20.png new file mode 100644 index 0000000000..f82135575b Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-creation-20.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-creation-30.png b/docs/docs/10-users/60-compute/60-job/guides/job-creation-30.png new file mode 100644 index 0000000000..fc20e02c12 Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-creation-30.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-creation-40.png b/docs/docs/10-users/60-compute/60-job/guides/job-creation-40.png new file mode 100644 index 0000000000..333a7d79fb Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-creation-40.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-10.png b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-10.png new file mode 100644 index 0000000000..eee6ce1b40 Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-10.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-20.png b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-20.png new file mode 100644 index 0000000000..bc0d5cc8bc Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-20.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-30.png b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-30.png new file mode 100644 index 0000000000..3365eec5f2 Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-30.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-40.png b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-40.png new file mode 100644 index 0000000000..26f0df4b32 Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-40.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-50.png b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-50.png new file mode 100644 index 0000000000..229f3d24dc Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-monitoring-50.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-submission-10.png b/docs/docs/10-users/60-compute/60-job/guides/job-submission-10.png new file mode 100644 index 0000000000..e2ff2d396d Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-submission-10.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-submission-20.png b/docs/docs/10-users/60-compute/60-job/guides/job-submission-20.png new file mode 100644 index 0000000000..ee2bca8968 Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-submission-20.png differ diff --git a/docs/docs/10-users/60-compute/60-job/guides/job-submission-30.png b/docs/docs/10-users/60-compute/60-job/guides/job-submission-30.png new file mode 100644 index 0000000000..72219f9650 Binary files /dev/null and b/docs/docs/10-users/60-compute/60-job/guides/job-submission-30.png differ diff --git a/docs/docs/10-users/60-compute/60-job/index.md b/docs/docs/10-users/60-compute/60-job/index.md new file mode 100644 index 0000000000..d57f78b368 --- /dev/null +++ b/docs/docs/10-users/60-compute/60-job/index.md @@ -0,0 +1,29 @@ +# Jobs + +A Renku job is a non-interactive run that executes a command in the background using a project's [Environment](../environment) and compute resources. + +A job is submitted from a **Job launcher** (see [Launcher](../launcher)) and runs without a browser-based interface. Inside the job, all of the project's [Code repositories](../../code/code-repository) are cloned and [Data connectors](../../data/data) are mounted, as long as you provided any required access credentials when submitting. + +Each job submission is identified by a **submission ID** — a short name you choose (or accept from the auto-generated default) that must be unique among your jobs on that launcher. You use the submission ID to tell jobs apart when several are running at once. + +A job has access to compute resources (CPU, GPU, RAM, and storage) determined by the resource class you select at submit time. For more information, see [Resource Pools & Classes](../resource-pools-and-classes). + +The jobs you submit are always private to you. Jobs are not shared between users. + +## Job lifecycle + +A job moves through states such as **Starting**, **Running**, **Completed**, and **Errored**. While a job is starting or running, you can view its logs or cancel it. When a job has completed or errored, you can view its logs and then **Dismiss** it to remove it from your project and dashboard. + +Dismissing a job permanently removes it from RenkuLab. You will not be able to reopen its logs after dismissing it. + +:::note + +Completed and errored jobs are typically removed automatically after 24 hours if you have not dismissed them. + +::: + +## When to use a Job + +Use a Job when you want to run a script or batch process in the background — for example, model training, data processing, or any long-running command that does not require an interactive notebook or editor. + +If you need an interactive environment for exploration or development, use a [Session](../session) instead. See [How to handle long training runs](../job/guides/handle-long-training-runs) if you are deciding between a Job and workarounds inside an interactive session. diff --git a/docs/docs/10-users/60-compute/70-app/_category_.yml b/docs/docs/10-users/60-compute/70-app/_category_.yml new file mode 100644 index 0000000000..cdd1bb956a --- /dev/null +++ b/docs/docs/10-users/60-compute/70-app/_category_.yml @@ -0,0 +1 @@ +label: Apps diff --git a/docs/docs/10-users/60-sessions/guides/99-set-up-shiny-app-launcher.md b/docs/docs/10-users/60-compute/70-app/guides/10-set-up-shiny-app-launcher.md similarity index 82% rename from docs/docs/10-users/60-sessions/guides/99-set-up-shiny-app-launcher.md rename to docs/docs/10-users/60-compute/70-app/guides/10-set-up-shiny-app-launcher.md index 0f8a05f3b5..57657ad707 100755 --- a/docs/docs/10-users/60-sessions/guides/99-set-up-shiny-app-launcher.md +++ b/docs/docs/10-users/60-compute/70-app/guides/10-set-up-shiny-app-launcher.md @@ -6,7 +6,7 @@ This how-to guide will configure a Shiny app to run in a Renku session (using R ::: -1. Add a session launcher to your Renku project with the following **Custom Environment** option as explained in [How to use your own docker image for a Renku session](/docs/users/sessions/guides/environments/use-your-own-docker-image-for-renku-session) considering the following parameters: +1. Add a session launcher to your Renku project with the following **Custom Environment** option as explained in [How to use your own docker image for a Renku session](../../environment/guides/use-your-own-docker-image-for-renku-session) considering the following parameters: 1. **Container Image:** [`http://registry.renkulab.io/bethcg/shiny-launcher-project:acb9b74`](http://registry.renkulab.io/bethcg/shiny-launcher:b710b82) 2. **Default URL:** `/shiny/work/<*path-to-app>*/` diff --git a/docs/docs/10-users/60-sessions/guides/_90-create-custom-environment-and-launcher-for-shiny-app.md b/docs/docs/10-users/60-compute/70-app/guides/20-create-custom-environment-and-launcher-for-shiny-app.md similarity index 72% rename from docs/docs/10-users/60-sessions/guides/_90-create-custom-environment-and-launcher-for-shiny-app.md rename to docs/docs/10-users/60-compute/70-app/guides/20-create-custom-environment-and-launcher-for-shiny-app.md index ea0fccbfaf..bb2ffd26bd 100755 --- a/docs/docs/10-users/60-sessions/guides/_90-create-custom-environment-and-launcher-for-shiny-app.md +++ b/docs/docs/10-users/60-compute/70-app/guides/20-create-custom-environment-and-launcher-for-shiny-app.md @@ -4,7 +4,7 @@ If you have a Shiny app that needs additional R packages, this how-to guide will help you create a new Renku environment that includes all packages and can run the Shiny app in the browser (using R version 4.2.0). -If you have a shiny app that does not need any additional R packages, you can instead use the simpler guide [How to set up a Shiny app launcher](/docs/users/sessions/guides/set-up-shiny-app-launcher). +If you have a shiny app that does not need any additional R packages, you can instead use the simpler guide [How to set up a Shiny app launcher](set-up-shiny-app-launcher). ::: @@ -16,26 +16,26 @@ We know this how-to guide is not easy, and we are hard at work to make this easi ### Attach your Shiny app code repository to your project: -1. [How to add a code repository to your project](/docs/users/code/guides/add-code-repository-to-project) +1. [How to add a code repository to your project](../../../code/guides/add-code-repository-to-project) ### Create a custom Renku environment and launcher to serve your Shiny app: 1. Create a fork of this repository: [https://gitlab.renkulab.io/bethcg/shiny-launcher](https://gitlab.renkulab.io/bethcg/shiny-launcher), we will refer to the fork as _project-fork_. -  +  -2. In _project-fork_, modify the `install.R` file to include the required R packages for your project, as explained [here](/docs/users/sessions/guides/environments/install-packages-on-the-fly-in-your-session#r). +2. In _project-fork_, modify the `install.R` file to include the required R packages for your project, as explained [here](../../session/guides/install-packages-on-the-fly-in-your-session#r). 3. Commit your changes for _project-fork_. 4. Select **Deploy(4a)/Container Registry(4b)** on the left menu of _project-fork_. 5. Click on top of your image registry to access the tags **(5)**. -  +  6. Copy the last tagged image (_registry_link_). -  +  -7. Add a session launcher to your Renku project with the following **Custom Environment** option as explained in [How to use your own docker image for a Renku session](/docs/users/sessions/guides/environments/use-your-own-docker-image-for-renku-session) considering the following parameters: +7. Add a session launcher to your Renku project with the following **Custom Environment** option as explained in [How to use your own docker image for a Renku session](../../environment/guides/use-your-own-docker-image-for-renku-session) considering the following parameters: 1. **Container Image:** `*user_registry_link*` 2. **Default URL:** `/shiny/work/<*path-to-app>*/` diff --git a/docs/docs/10-users/60-compute/70-app/guides/30-create-streamlit-app.md b/docs/docs/10-users/60-compute/70-app/guides/30-create-streamlit-app.md new file mode 100644 index 0000000000..d5e284c1d5 --- /dev/null +++ b/docs/docs/10-users/60-compute/70-app/guides/30-create-streamlit-app.md @@ -0,0 +1,50 @@ +# How to create a Streamlit app on top of your project + +:::info + +This documentation is out-of-date and will be updated soon! + +::: + +As of now, it is still possible to launch your Streamlit app on top of your Renku 2.0 project as follows. + +1. Create a fork of this repository: https://gitlab.renkulab.io/rok.roskar/streamlit-v2-example, we will refer to the fork as project-fork. + ++ +
+ +2. In _project-fork_, modify line 8 in the file `jupyter_noteboox_config.py`to the path to your python entry file for Streamlit within your Renku 2.0 project ( e.g. `work/name-fork-project/helloworld.py` or `work/main-project-repository/streamlit/app.py` or `work/repository-2/app.py` ) + + ⚠️ Do not forget to add `work/` at the beginning of the path ! + ++ +
+ +3. Add all the package dependencies you will need to run your Renku 2.0 project in the requirements.txt file or in the environment.yml , as explained [here](../../session/guides/install-packages-on-the-fly-in-your-session). + ++ +
+ +4. Commit your changes for _project-fork_. +5. Select **Deploy(5a)/Container Registry(5b)** on the left menu of _project-fork_. +6. Click on top of your image registry to access the tags **(6)**. + ++ +
+ +7. Copy the latest tagged image (registry_link). + ++ +
+ +8. Add a session launcher to your Renku 2.0 project the following **Custom Environment** set-up \*\*\*\*as described in [How to use your own docker image for a Renku session](../../environment/guides/use-your-own-docker-image-for-renku-session). Consider the following two parameters: + 1. **Container Image:** `*registry_link*` + 2. **Default URL:** `/streamlit` +9. [Optional] If the entry point for your streamlit app will be hosted in _project-fork,_ add _project-fork_ code repository to your Renku 2.0 project. + +You can now start a session with your new streamlit launcher to access the streamlit app directly in your browser. diff --git a/docs/docs/10-users/60-sessions/guides/_80-create-dashboard.md b/docs/docs/10-users/60-compute/70-app/guides/_80-create-dashboard.md similarity index 68% rename from docs/docs/10-users/60-sessions/guides/_80-create-dashboard.md rename to docs/docs/10-users/60-compute/70-app/guides/_80-create-dashboard.md index 1116ba4d1c..11ca4e0974 100755 --- a/docs/docs/10-users/60-sessions/guides/_80-create-dashboard.md +++ b/docs/docs/10-users/60-compute/70-app/guides/_80-create-dashboard.md @@ -1,12 +1,12 @@ # How to create a Dashboard (Streamlit, Plotly Dash) -Renku allows Docker images with user-specific entry points. This feature makes it easy the creation of user-specific environments, including setting up Streamlit apps. Check [How to use your own docker image for a Renku session](/docs/users/sessions/guides/environments/use-your-own-docker-image-for-renku-session) for the full list of details and options. +Renku allows Docker images with user-specific entry points. This feature makes it easy the creation of user-specific environments, including setting up Streamlit apps. Check [How to use your own docker image for a Renku session](../../environment/guides/use-your-own-docker-image-for-renku-session) for the full list of details and options. -There are mainly two options two have your own Streamlit app served in Renku, either without access to the codebase or with access to the code. +There are mainly two options to have your own Streamlit app served in Renku, either without access to the codebase or with access to the code. # Create an app with a code-based environment -Check [How to create an environment with custom packages installed](/docs/users/sessions/guides/environments/create-environment-with-custom-packages-installed) for the full list of details. +Check [How to create an environment with custom packages installed](../../environment/guides/create-environment-with-custom-packages-installed) for the full list of details. :::note @@ -18,19 +18,19 @@ Your project might have a nice dashboard inside, which you would want others to To set up a dashboard with an environment built from your repository, you can follow these steps: -1. Follow the steps for creating a [code-based environment](/docs/users/sessions/guides/environments/create-environment-with-custom-packages-installed) above. +1. Follow the steps for creating a [code-based environment](../../environment/guides/create-environment-with-custom-packages-installed) above. 2. Once the image is done building, edit the environment and change it to a “Custom Environment” -3. Edit the `Command` to be `["bash", "-c"]` and `Args` to correspond to your app - see common examples [here](/docs/users/sessions/guides/environments/use-your-own-docker-image-for-renku-session). +3. Edit the `Command` to be `["bash", "-c"]` and `Args` to correspond to your app - see common examples [here](../../environment/guides/use-your-own-docker-image-for-renku-session). Once you are done, your environment configuration should look something like this:- +
And your launcher set up could be, for example: - + # 🔒Special Use Case: Create an app without sharing the code @@ -46,7 +46,7 @@ Create a session launcher using the **Custom Image** option, and provide the fol ["sh", "-c"] ``` -- Command Arguments CMD (fill in ` diff --git a/docs/docs/10-users/60-compute/90-guides/_category_.yml b/docs/docs/10-users/60-compute/90-guides/_category_.yml new file mode 100644 index 0000000000..8cca0baa51 --- /dev/null +++ b/docs/docs/10-users/60-compute/90-guides/_category_.yml @@ -0,0 +1,2 @@ +label: How to... +collapsed: false diff --git a/docs/docs/10-users/60-sessions/guides/configure-session-secret-10.png b/docs/docs/10-users/60-compute/90-guides/configure-session-secret-10.png similarity index 100% rename from docs/docs/10-users/60-sessions/guides/configure-session-secret-10.png rename to docs/docs/10-users/60-compute/90-guides/configure-session-secret-10.png diff --git a/docs/docs/10-users/60-sessions/guides/configure-session-secret-20.png b/docs/docs/10-users/60-compute/90-guides/configure-session-secret-20.png similarity index 100% rename from docs/docs/10-users/60-sessions/guides/configure-session-secret-20.png rename to docs/docs/10-users/60-compute/90-guides/configure-session-secret-20.png diff --git a/docs/docs/10-users/60-sessions/guides/configure-session-secret-30.png b/docs/docs/10-users/60-compute/90-guides/configure-session-secret-30.png similarity index 100% rename from docs/docs/10-users/60-sessions/guides/configure-session-secret-30.png rename to docs/docs/10-users/60-compute/90-guides/configure-session-secret-30.png diff --git a/docs/docs/10-users/60-sessions/guides/configure-session-secret-40.png b/docs/docs/10-users/60-compute/90-guides/configure-session-secret-40.png similarity index 100% rename from docs/docs/10-users/60-sessions/guides/configure-session-secret-40.png rename to docs/docs/10-users/60-compute/90-guides/configure-session-secret-40.png diff --git a/docs/docs/10-users/60-sessions/guides/configure-session-secret-50.png b/docs/docs/10-users/60-compute/90-guides/configure-session-secret-50.png similarity index 100% rename from docs/docs/10-users/60-sessions/guides/configure-session-secret-50.png rename to docs/docs/10-users/60-compute/90-guides/configure-session-secret-50.png diff --git a/docs/docs/10-users/60-sessions/guides/select-compute-resources-for-session-10.mov b/docs/docs/10-users/60-compute/90-guides/select-compute-resources-for-session-10.mov similarity index 100% rename from docs/docs/10-users/60-sessions/guides/select-compute-resources-for-session-10.mov rename to docs/docs/10-users/60-compute/90-guides/select-compute-resources-for-session-10.mov diff --git a/docs/docs/10-users/60-compute/_category_.yml b/docs/docs/10-users/60-compute/_category_.yml new file mode 100644 index 0000000000..edb8a56e83 --- /dev/null +++ b/docs/docs/10-users/60-compute/_category_.yml @@ -0,0 +1 @@ +label: Launchers & Compute diff --git a/docs/docs/10-users/60-compute/launcher-10.png b/docs/docs/10-users/60-compute/launcher-10.png new file mode 100644 index 0000000000..abc6a50428 Binary files /dev/null and b/docs/docs/10-users/60-compute/launcher-10.png differ diff --git a/docs/docs/10-users/60-sessions/resource-pools-and-classes-10.png b/docs/docs/10-users/60-compute/resource-pools-and-classes-10.png similarity index 100% rename from docs/docs/10-users/60-sessions/resource-pools-and-classes-10.png rename to docs/docs/10-users/60-compute/resource-pools-and-classes-10.png diff --git a/docs/docs/10-users/60-sessions/resource-pools-and-classes-20.jpg b/docs/docs/10-users/60-compute/resource-pools-and-classes-20.jpg similarity index 100% rename from docs/docs/10-users/60-sessions/resource-pools-and-classes-20.jpg rename to docs/docs/10-users/60-compute/resource-pools-and-classes-20.jpg diff --git a/docs/docs/10-users/60-sessions/10-environment.md b/docs/docs/10-users/60-sessions/10-environment.md deleted file mode 100755 index 0c1bed8341..0000000000 --- a/docs/docs/10-users/60-sessions/10-environment.md +++ /dev/null @@ -1,12 +0,0 @@ -# Environment - -An environment is a docker-based environment that can run in a RenkuLab interactive [Session](session) . - -An environment consists of: - -1. a reference to a docker image -2. the configuration required to run that docker image on RenkuLab - -Renku provides several _global environments_ that come pre-configured and ready to use. Alternatively, you can create your own custom environment with your own docker image. - -Environments are connected to a Renku project via a [Session Launcher](session-launcher). diff --git a/docs/docs/10-users/60-sessions/20-session-launcher.md b/docs/docs/10-users/60-sessions/20-session-launcher.md deleted file mode 100755 index 02630bceb2..0000000000 --- a/docs/docs/10-users/60-sessions/20-session-launcher.md +++ /dev/null @@ -1,12 +0,0 @@ -# Session Launcher - -A session launcher is a component of a project that launches an interactive, browser-based compute [Session](session). - -A session launcher contains: - -1. a session [Environment](environment), which defines the software installed in the session -2. a default resource pool (see [Resource Pools & Classes](resource-pools-and-classes)), which determines the compute resources available in the session - -A project’s session launchers are usable to everyone who can see the project. This means that you can configure a session launcher with an [Environment](environment) , and everyone in the project can use that session launcher to launch a session with the exact same environment. - -However, the sessions you launch are only accessible to you. Sessions are not shared between users. diff --git a/docs/docs/10-users/60-sessions/25-session.md b/docs/docs/10-users/60-sessions/25-session.md deleted file mode 100755 index ae18a33a37..0000000000 --- a/docs/docs/10-users/60-sessions/25-session.md +++ /dev/null @@ -1,9 +0,0 @@ -# Session - -A Renku session is a running interactive run-time environment that can be used to do work on data and code. - -A session is created by a [Session Launcher](session-launcher) and is a running instance of an [Environment](environment). Inside a session, all of the project’s [Code repository](../code/code-repository)s are cloned and [Data connector](../data/data)s are mounted (as long as the user provided any required access credentials). - -A session has access to a certain amount of compute resources (CPU, GPU, RAM, and storage). This is determined by the resource pool set on the [Session Launcher](session-launcher). For more information about compute resources, see [Resource Pools & Classes](resource-pools-and-classes). - -The sessions you launch are always private to you. diff --git a/docs/docs/10-users/60-sessions/30-resource-pools-and-classes.md b/docs/docs/10-users/60-sessions/30-resource-pools-and-classes.md deleted file mode 100755 index d00921574f..0000000000 --- a/docs/docs/10-users/60-sessions/30-resource-pools-and-classes.md +++ /dev/null @@ -1,38 +0,0 @@ -# Resource Pools & Classes - -The compute resource available to Renku users are determined by the **resources pools** the user has access to. To specify the resources for a project’s session launcher, you select a **resource class** from any of the resource pools available to you. - -## Resource Pools - -A **resource pool** is a total resource quota, a set of users who are allowed to use resources from that quota, and culling times for sessions using those resources. - -- **Resources:** For example, a resource pool may be configured for 160 CPUs, 1024 GB RAM, and 16 GPUs. -- **Users:** There is no limit to how many users can be added to a resource pool. And conversely, a user may be granted access to any number of resource pools. All resource pools the user has access to show up in their resource selector when they configure their RenkuLab session. -- **Culling times:** You can customize the max idle time for your resource pool(s). For example, in the Public resource pool (the default resource pool available to all RenkuLab users), sessions auto-pause after 2 hours of idle time, and shut down after 2 weeks. You may customize these thresholds to be longer or shorter. - -## Resource Classes - -A resource pool contains a set of **resource classes**. A resource class is a resource configuration that a user can select for their RenkuLab session. - -For example, a resource class may be: - -- 12 CPUs, 80 GB RAM, 2 GPUs, 50 GB _default_ disk, 500 GB _max_ disk - -As users run sessions, their session resources (the resource class the session is using) comes out of the quota on the resource pool. If the resource pool’s quota has been reached, users won’t be able to start any more sessions in that resource pool (but they may run sessions in other resource pools). - -
- -*For custom resource pools granted to users, you may customize the resource classes in you resource pool.* -
- -## The Public Resource Pool & Classes - -On RenkuLab, the following resources classes are publicly available (in the “public resource pool”): - -- -
- -## Request a Custom Resource Pool {#request-custom-resource-pool} - -Do you need more resources than are available in RenkuLab’s public resource classes? [Contact](../community) us! We can configure a custom resource pool for your needs upon demand. We apply flat, transparent pricing, passing down infra rates savings. diff --git a/docs/docs/10-users/60-sessions/_category_.yml b/docs/docs/10-users/60-sessions/_category_.yml deleted file mode 100644 index a9f02bb94f..0000000000 --- a/docs/docs/10-users/60-sessions/_category_.yml +++ /dev/null @@ -1 +0,0 @@ -label: Compute Sessions diff --git a/docs/docs/10-users/60-sessions/guides/00-environments/45-use-your-own-docker-image-for-renku-session.md b/docs/docs/10-users/60-sessions/guides/00-environments/45-use-your-own-docker-image-for-renku-session.md deleted file mode 100755 index df7dcd4dd6..0000000000 --- a/docs/docs/10-users/60-sessions/guides/00-environments/45-use-your-own-docker-image-for-renku-session.md +++ /dev/null @@ -1,224 +0,0 @@ -# Use your own docker image for a Renku session - -## What Docker images can I use in RenkuLab sessions? - -There are some limitations to what images you can use in Renku sessions. The image must meet **all** of the conditions below in order to work on RenkuLab: - -1. **The image needs a front end** - - The image you use needs a user interface web frontend to run so that you can access the session via the browser. Examples of these include JupyterLab, VSCode or RStudio. - -2. **The image must be public** - - RenkuLab can not yet use images that not publicly accessible. - -3. **The image will be run as non-root** - - For security reasons, sessions are started with a non-root user. - -## How to configure an image as a custom environment - -In the project page: - -1. Under **Sessions** section click on ➕ to add a new launcher -2. Select **External environment** - --  -
-3. For the container image, provide an **image identifier**. - - Some examples of image identifiers: - - if the image is hosted on DockerHub: - - `renku/renkulab-py:3.10-0.24.0` - - `continuumio/anaconda3:2024.06-1` - - if the image is hosted on GitHub: - - `ghcr.io/swissdatasciencecenter/renku-frontend-buildpacks/base-image:0.0.8` - - The image identifier should be in the format that works with `docker pull` -4. If needed, fill in the **Advanced settings** to tell Renku how to run your image. The available fields are: - - **Container Image**: the image identifier to use (e.g. `renku/renkulab-py:3.10-0.24.0`) - - **Default URL**: the path appended to the session URL to open the front end (e.g. `/lab` for JupyterLab, `/rstudio` for RStudio) - - **Mount Directory**: the directory inside the container where Renku mounts your project files (e.g. `/home/jovyan/work`) - - **Working Directory**: the directory the session opens in (e.g. `/home/jovyan/work`) - - **UID**: the user ID the container process runs as (e.g. `1000`) - - **GID**: the group ID the container process runs as (e.g. `100`) - - **Command ENTRYPOINT**: the shell entrypoint (e.g. `["sh", "-c"]`) - - **Command Arguments**: the command passed to the entrypoint to start the front end server - See [Example image configurations for common front ends](#example-image-configurations) below for ready-made values per front end. -5. Select the **Resource class** that best fits your expected computational needs. - :::tip - If the available resource classes are too small for your compute requirements, we can create a custom resource pool for you! See [Request a Custom Resource Pool](../../resource-pools-and-classes#request-custom-resource-pool). - ::: - -6. Give your session launcher a **name**. -7. Click on **Add session launcher** button. - -:::info -Note that you can always **modify your session launcher** by clicking on top of it on the project’s page, and using the menu on the right: -- -
-::: - -## Example image configurations for common front ends {#example-image-configurations} - -In order to run a docker image in a session, Renku needs to know some information about how to run and serve that image. - -In this section, you can see example configurations for commonly used images. If you build an image with one of these images as the base, then you can use this provided configuration to make that image run in RenkuLab. The information below can be copied and pasted into the **Advanced Settings** form for creating a **custom environment**. - -- -
- -### Jupyter - -- Container Image: `jupyter/minimal-notebook:python-3.11` -- Port: `8888` -- Default URL: `/lab` -- Command ENTRYPOINT: - -```json -["sh", "-c"] -``` - -- Command Arguments CMD ([learn more](#about-renku-session-urls)): - -```json -[ - "jupyter server --ServerApp.ip=0.0.0.0 --ServerApp.port=8888 --ServerApp.base_url=$RENKU_BASE_URL_PATH --ServerApp.token=\"\" --ServerApp.password=\"\" --ServerApp.allow_remote_access=true --ContentsManager.allow_hidden=true --ServerApp.allow_origin=*" -] -``` - -### Julia and Jupyter - -- Container Image: `jupyter/julia-notebook:x86_64-python-3.11.6` -- Port: `8888` -- Default URL: `/lab` -- Working Directory: `/home/jovyan/work` -- Mount Directory: `/home/jovyan/work` -- Command ENTRYPOINT: - -```json -["sh", "-c"] -``` - -- Command Arguments CMD ([learn more](#about-renku-session-urls)): - -```json -[ - "jupyter server --ServerApp.ip=0.0.0.0 --ServerApp.port=8888 --ServerApp.base_url=$RENKU_BASE_URL_PATH --ServerApp.token=\"\" --ServerApp.password=\"\" --ServerApp.allow_remote_access=true --ContentsManager.allow_hidden=true --ServerApp.allow_origin=*" -] -``` - -### Rstudio - -🚧 _Not yet available_ - -### RShiny - -🚧 _Not yet available_ - -### VSCode - -- Container Image: Build a docker image that includes vscode in the PATH -- Port: `8888` -- Command ENTRYPOINT: - -```json -["sh", "-c"] -``` - -- Command Arguments CMD ([learn more](#about-renku-session-urls)): - -```json -[ - "code serve-web --server-base-path $RENKU_BASE_URL_PATH/ --without-connection-token --host 0.0.0.0 --port 8888" -] -``` - -### Streamlit - -- Container Image: Build a docker image that includes streamlit and any other requirements needed by your streamlit app -- Port: `8888` -- Command ENTRYPOINT: - -```json -["sh", "-c"] -``` - -- Command Arguments CMD (fill in `
-Rather than hosting all these resources itself, Renku is a connecting hub that provides the means to bring them all together. For example, a project might use a published dataset from [Zenodo](https://www.notion.so/How-to-connect-data-from-data-repositories-such-as-Zenodo-or-Dataverse-1eb0df2efafc802ab3bef1c47c8c45b4?pvs=21), raw data from a shared [cloud storage resource](/docs/users/data/data) and code from a [GitHub repository](/docs/users/code/guides/add-code-repository-to-project). These are then made seamlessly available in containerized browser-based compute [sessions](/docs/users/sessions/session). Renku sessions can run on Renku servers or on an external computational resources such as a local cloud provider or - coming soon! - an HPC cluster. The sessions can be used for development with [environments](/docs/users/sessions/guides/environments) like VSCode, Jupyter, or RStudio, or they can be used to display dashboards with Streamlit or Shiny. To promote sustainable computational research, Renku also makes it straightforward to build Docker images based on users' [software requirements](/docs/users/sessions/guides/environments/create-environment-with-custom-packages-installed), ensuring that everyone using a project does so with the right software stack. +Rather than hosting all these resources itself, Renku is a connecting hub that provides the means to bring them all together. For example, a project might use a published dataset from [Zenodo](https://www.notion.so/How-to-connect-data-from-data-repositories-such-as-Zenodo-or-Dataverse-1eb0df2efafc802ab3bef1c47c8c45b4?pvs=21), raw data from a shared [cloud storage resource](/docs/users/data/data) and code from a [GitHub repository](/docs/users/code/guides/add-code-repository-to-project). These are then made seamlessly available in containerized browser-based compute [sessions](/docs/users/compute/session). Renku sessions can run on Renku servers or on an external computational resources such as a local cloud provider or - coming soon! - an HPC cluster. The sessions can be used for development with [environments](/docs/users/compute/environment) like VSCode, Jupyter, or RStudio, or they can be used to display dashboards with Streamlit or Shiny. To promote sustainable computational research, Renku also makes it straightforward to build Docker images based on users' [software requirements](/docs/users/compute/environment/guides/create-environment-with-custom-packages-installed), ensuring that everyone using a project does so with the right software stack.
-Collaborate on a fully reproducible Renku project by adding project [members](/docs/users/collaboration/guides/add-people-to-project), and showcase interactive demos and apps with anyone via [public session launch links](/docs/users/sessions/guides/share-session-launch-link). For larger collaborations, users can organize projects and other assets under [groups](/docs/users/use-cases/groups), which helps streamline collaboration. +Collaborate on a fully reproducible Renku project by adding project [members](/docs/users/collaboration/guides/add-people-to-project), and showcase interactive demos and apps with anyone via [public session launch links](/docs/users/compute/session/guides/share-session-launch-link). For larger collaborations, users can organize projects and other assets under [groups](/docs/users/use-cases/groups), which helps streamline collaboration. Renku is developed under the Apache 2.0 license and can be deployed on a variety of cloud infrastructures offering Kubernetes. The flagship instance operated by the Swiss Data Science Center is available under [https://renkulab.io](https://renkulab.io/) and is open to everyone with a publicly-accessible free resource tier.