Merge pull request #4812 from mathesar-foundation/release-0.6.0

docker-compose.dev.yml

@@ -65,6 +65,7 @@ services:
       - ALLOWED_HOSTS=${ALLOWED_HOSTS-*}
       - SECRET_KEY=${SECRET_KEY-}
       - OIDC_CONFIG_DICT=${OIDC_CONFIG_DICT-{}}
+      - FILE_STORAGE_DICT=${FILE_STORAGE_DICT-{}}
       - MATHESAR_ANALYTICS_URL=${MATHESAR_ANALYTICS_URL-https://example.com/collector}
       - MATHESAR_INIT_REPORT_URL=${MATHESAR_INIT_REPORT_URL-https://example.com/hello}
       - MATHESAR_FEEDBACK_URL=${MATHESAR_FEEDBACK_URL-https://example.com/feedback}

docker-compose.yml

@@ -56,6 +56,10 @@ x-config: &config
   # (Optional) Replace '5432' with the port on which postgres is running
   POSTGRES_PORT: ${POSTGRES_PORT:-5432}
 
+  # (Optional) Can be used to configure an S3-compliant file storage backend
+  # instead of a file_storage.yml configuration file.
+  FILE_STORAGE_DICT: ${FILE_STORAGE_DICT:-{}}
+
   # (Optional) Can be used to configure single-sign on (sso) instead of an
   # sso.yml configuration file.
   # See https://docs.mathesar.org/administration/single-sign-on/ for additional
@@ -105,6 +109,11 @@ x-config: &config
 #           client applications.
 #     Example: 5555
 #
+# FILE_STORAGE_DICT:
+#     Default: {}
+#     Info: Specifies the settings necessary to configure an S3-complaint
+#           filestorage backend.
+#
 # OIDC_CONFIG_DICT:
 #   Default: {}
 #   Info: Specifies the settings necessary to configure Single Sign-on (SS0)
@@ -168,10 +177,20 @@ services:
       # leave this as-is.
       SECRET_KEY: ${SECRET_KEY-}
 
+      # If you are running Mathesar on a particularly powerful machine, you can
+      # improve performance by increasing this setting. Replace '3' below with
+      # 2 * $(NUM_CORES) + 1. So if you have a VPS with 4 vCPUs you can use a
+      # value of 2 * 4 + 1, or 9. The default is workable as long as you have at
+      # least 1 logical core.
+      WEB_CONCURRENCY: ${WEB_CONCURRENCY-3}
+
     volumes:
       - ./msar/static:/code/static
       - ./msar/media:/code/media
       - ./msar/secrets:/code/.secrets
+      # Uncomment the following to mount file_storage.yml and enable
+      #  an S3-compliant file storage backend
+      # - ./file_storage.yml:/code/file_storage.yml
       # Uncomment the following to mount sso.yml and enable Single Sign-On (SSO).
       # - ./sso.yml:/code/sso.yml
     depends_on:

docs/docs/administration/environment-variables.md

@@ -21,6 +21,12 @@ This page contains all available environment variables supported by Mathesar. Se
 - **Format**: A set of host/domain names separated by a comma
 - **Default value**: `.localhost,127.0.0.1,[::1]`
 
+### `WEB_CONCURRENCY` {: #web_concurrency}
+
+- **Description**: Sets the number of sync Gunicorn workers, affecting the number of concurrent requests Mathesar can handle. Bigger is better, subject to system resources. The typically-recommended number is `2 * $(NUM_PROC) + 1`, where `NUM_PROC` is the number of logical cores on your machine. So, if Mathesar is running on a server with 4 vCPUs, then this should be set to 9.
+- **Format**: An integer.
+- **Default value**: `3`
+
 
 ## Internal database configuration {: #db}
 
@@ -101,3 +107,22 @@ The database specified in this section is used to store Mathesar's internal data
 - **Additional information**: The following tools might help you convert the YAML syntax from `sso.yml` into the proper format:
     - [Convert YAML to JSON](https://onlineyamltools.com/convert-yaml-to-json)
     - [JSON stringify online](https://jsonformatter.org/json-stringify-online)
+
+## File backend configuration
+
+!!! info "**OPTIONAL**"
+    Only needed if [using file columns](../user-guide/files.md), in installations where the local filesystem is inaccessible.
+
+### `FILE_STORAGE_DICT` (optional)
+
+- **Description**: The configuration to connect Mathesar to an S3-compatible [file storage backend](../administration/file-backend-config.md).
+- **Format**: A stringified JSON representation of the config in the [`file_storage.yml` file](https://github.com/mathesar-foundation/mathesar/raw/{{mathesar_version}}/file_storage.yml.example).
+
+    !!! example
+        ```env
+         FILE_STORAGE_DICT="{\"default\":{\"protocol\":\"s3\",\"nickname\":\"Example\",\"prefix\":\"mathesar-storage\",\"kwargs\":{\"client_kwargs\":{\"endpoint_url\":\"https:\/\/storage-example.mathesar.org\",\"region_name\":\"auto\",\"aws_access_key_id\":\"XXX\",\"aws_secret_access_key\":\"XXX\"}}}}"
+        ```
+
+- **Additional information**: The following tools might help you convert the YAML syntax from `file_storage.yml` into the proper format:
+    - [Convert YAML to JSON](https://onlineyamltools.com/convert-yaml-to-json)
+    - [JSON stringify online](https://jsonformatter.org/json-stringify-online)

docs/docs/administration/file-backend-config.md

@@ -0,0 +1,158 @@
+# Configuring file storage backends
+
+!!! question "Help us refine file storage!"
+    Our file storage feature is new and still evolving. We'd love to hear about how you're using it, what's working, and what additional workflows you'd like to see supported.
+
+	[Talk to us for 20 min](https://cal.com/mathesar/users)! We'll give you a $25 gift card as a thank you.
+
+Mathesar's [**file data type**](../user-guide/files.md) requires you to configure an **S3-compatible object storage backend**. File storage allows users to upload, preview, and download files directly within Mathesar.
+
+**Configuring file storage and using file columns is optional.** By default, Mathesar does not expose file column controls unless a storage backend is set up. If you do not set up a backend, users will not be able to work directly with files in Mathesar.
+
+## 1. Configure your storage backend
+
+First, you'll need to decide which **S3-compatible object storage backend** you'll use.
+
+Popular options include:
+
+- [AWS S3](https://docs.aws.amazon.com/s3/)
+- [Cloudflare R2](https://www.cloudflare.com/developer-platform/products/r2/)
+- [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces)
+- [Self-hosted MinIO](https://www.min.io/)
+
+!!! warning "MinIO license considerations"
+    MinIO is licensed under the [GNU Affero General Public License v3 (AGPLv3)](https://github.com/minio/minio/blob/master/LICENSE). This license requires that if you modify MinIO and make it available over a network, you must also make your modified source code available to users.
+
+    Be sure you understand these obligations before choosing MinIO as your storage backend. See MinIO's documentation on [license compliance](https://github.com/minio/minio/blob/master/COMPLIANCE.md#agplv3-compliance) for details.
+
+After choosing a backend, you'll need to follow the basic steps for working with S3-comptiable object storage:
+
+1. **Create a bucket**
+    - Decide on a unique bucket name.
+    - Choose which region your bucket is in (if your platform supports regions).
+    - (Optional but recommended) enable versioning or lifecycle policies depending on your retention needs.
+1. **Create an API key and secret key**
+    - Generate credentials with **read/write permissions** scoped to your bucket.
+2. **Note the endpoint URL**
+    - For example, AWS uses `https://s3.[region].amazonaws.com` by default.
+    - Other providers (Cloudflare R2, DigitalOcean Spaces, MinIO) each provide a specific endpoint.
+
+### 2. Enable file storage in Mathesar
+
+Enabling file storage requires setting up a new configuration file or environment variable that contains your backend's connection details.
+
+You'll either create a file named `file_storage.yml` or set up the `FILE_STORAGE_DICT` environment variable. Which you choose depends on how you installed Mathesar:
+
+=== "For Docker Compose installations"
+
+      If you used [Docker Compose](./install-via-docker-compose.md), create the `file_storage.yml` file next to your [`docker-compose.yml` file](https://github.com/mathesar-foundation/mathesar/raw/{{mathesar_version}}/docker-compose.yml):
+
+      ```diff
+      mathesar
+       ├── docker-compose.yml
+       ├── msar/
+      +└── file_storage.yml
+      ```
+
+      Then, uncomment the following lines in your docker compose file:
+
+      ```diff
+      volumes:
+       - ./msar/static:/code/static
+       - ./msar/media:/code/media
+      # Uncomment the following to mount file_storage.yml and enable
+      #  an S3-compliant file storage backend
+      -# - ./file_storage.yml:/code/file_storage.yml
+      +  - ./file_storage.yml:/code/file_storage.yml
+      ```
+
+=== "For Linux, macOS, or WSL installations"
+
+    For [direct installations](./install-from-scratch.md), create `file_storage.yml` in the installation directory you [defined while installing](./install-from-scratch.md#set-up-your-installation-directory) Mathesar.
+
+=== "For DigitalOcean or Railway"
+
+    If you deployed to an environment where you can't use the local filesystem (e.g. [DigitalOcean](./install-digitalocean.md), [Railway](./install-railway.md)), you can use the `FILE_STORAGE_DICT` environment variable instead of a `file_storage.yml` file.
+
+    This variable must contain a JSON representation of the same data that `file_storage.yml` contains, wrapped in quotes and with quotes escaped. Here's an example:
+
+    ```env
+    FILE_STORAGE_DICT="{\"default\": {\"protocol\": \"s3\", \"nickname\": \"Local object store\", \"prefix\": \"mathesar-storages\", \"kwargs\": {\"client_kwargs\": {\"endpoint_url\": \"http://mathesar-dev-store:9000\", \"region_name\": \"us-east-2\", \"aws_access_key_id\": \"mathesar\", \"aws_secret_access_key\": \"mathesar\"}}}}"
+    ```
+
+    !!! tip "See [Environment Variables](./environment-variables.md#file_storage_dict-optional) for links to tools that simplify this process."
+
+### 3. Configure backend connection details
+
+Here's where you populate your configuration file.
+
+#### 3a. Retrieve needed information
+
+From your chosen storage backend, collect the following details:
+
+* **Bucket name**: corresponds to the `prefix` field in Mathesar's config.
+* **Endpoint URL**: the base URL for your provider's S3 API.
+* **Region name**: required by some providers (e.g. AWS). If your provider doesn’t use regions (e.g. Cloudflare R2), set this to `"auto"`.
+* **Access key ID**: generated credential for programmatic access.
+* **Secret access key**: generated credential paired with the access key ID.
+
+#### 3b. Basic setup
+
+Once you have those details, update `file_storage.yml` with your values:
+
+```diff
+# This config file allows you to configure an S3-compatible
+# object storage backend for file columns in Mathesar.
+default:
++  protocol: s3
++  nickname: "Backend name"   # A friendly label for this backend
++  prefix: my-mathesar-bucket # This should match your bucket name exactly
++  kwargs:
++    client_kwargs:
++      endpoint_url: https://s3.us-east-2.amazonaws.com
++      region_name: us-east-2
++      aws_access_key_id: YOUR_ACCESS_KEY
++      aws_secret_access_key: YOUR_SECRET_KEY
+```
+
+### 4. Activate file storage
+
+Once you've finished configuring storage, restart Mathesar so it can load the updated `file_storage.yml` file or environment variable.
+
+=== "For Docker Compose installations"
+
+       ```bash
+       docker compose -f docker-compose.yml down
+       docker compose -f docker-compose.yml up -d
+       ```
+
+=== "For Linux, macOS, or WSL installations"
+
+    ```bash
+    sudo systemctl restart mathesar.service
+    ```
+
+=== "For DigitalOcean or Railway"
+
+    Use their web interface to restart (often framed as a "redeploy") Mathesar.
+
+After restarting, file columns will be enabled in your Mathesar installation. To test this, check the UI for adding a new file column.
+
+## Tips & technical information
+
+### How files are stored
+
+Files are stored in your PostgreSQL database in `JSONB` columns. A typical file value looks like this:
+
+```json
+{
+  "uri": "s3://my-mathesar-bucket/my-username/20250919-192215167015/example.csv",
+  "mash": "58f47a1eafd567cd9d0bdfa1f42a01978cc6f36eb7937b310b208d9957b7ee8b"
+}
+```
+
+With the `uri` being the path to the file on the storage backend and the mash being a generated, unique value used by Mathesar.
+
+### Removing file backends
+
+If you remove a file backend, you'll no longer see the "File" option in the "add column" dropdown. Existing file data will be preserved in your database, but in Mathesar's UI you'll see the underlying JSONB instead of image thumbnails or file icons.

docs/docs/administration/install-from-scratch.md

@@ -42,7 +42,7 @@ These steps create Mathesar's [internal database](../user-guide/databases.md#int
         Replace `strong‑pw‑here` with a strong, private password and make a note of it, you’ll need it later.
 
     !!! tip
-        About the extra privileges in the command,
+        About the extra privileges in the command:
 
         - **`CREATEDB`: gives the Postgres user permission to create new databases.**
             - It’s handy if you want Mathesar admins to spin up databases from the web interface.
@@ -202,6 +202,7 @@ Turn your local Mathesar installation into a public-facing production service.
     - Follow this section if you want Mathesar to run continuously on a server and be reachable by other users (with or without a public domain).
     - For personal use, evaluation, or on‑prem workstations, you can simply start Mathesar on demand with `mathesar run` and skip ahead to setting up your user account.
     - See our [guide to setting up single sign-on (SSO)](./single-sign-on.md) if that's of interest to you.
+    - Consider [configuring a file backend](./file-backend-config.md) to enable the file data type.
 
 !!! note "Linux-only"
     - The steps below **only target Linux servers** that use **systemd**.
@@ -211,6 +212,9 @@ Turn your local Mathesar installation into a public-facing production service.
 !!! note "Elevated permissions needed"
     Most of the commands below need to be run as a root user, or using `sudo`. If you try to run one of these commands, and see an error about "permission denied", run again with elevated privileges.
 
+!!! note "Gunicorn worker configuration"
+    By default Mathesar will use `3` Gunicorn sync workers. You may wish to adjust this if you're running Mathesar on a more powerful machine with additional vCPU cores. [See our recommendations for the `WEB_CONCURRENCY` env var to learn more](./environment-variables.md#web_concurrency).
+
 ### Run Mathesar as a systemd service
 
 These steps create a systemd service to run Mathesar continuously - 24x7.

docs/docs/administration/install-via-docker-compose.md

@@ -92,7 +92,7 @@ If you want Mathesar to be accessible over the internet, you'll probably want to
 
 **Ensure that the DNS for your domain or sub-domain is pointing to the public IP address of the machine that you're installing Mathesar on**.
 
-Add your domain(s) or sub-domain(s) to the [`DOMAIN_NAME`](../../configuration/env-variables/#domain_name) environment variable, in the **CONFIG** section of the docker-compose file.
+Add your domain(s) or sub-domain(s) to the [`DOMAIN_NAME`](./environment-variables.md#domain_name) environment variable, in the **CONFIG** section of the docker-compose file.
 
 !!! example
     ```yaml
@@ -101,6 +101,12 @@ Add your domain(s) or sub-domain(s) to the [`DOMAIN_NAME`](../../configuration/e
 
 Restart the docker containers for the configuration to take effect.
 
+### Modifying the number of Gunicorn workers
+
+If you're deploying Mathesar in a production or multi-user environment, you may want to increase the number of Gunicorn workers to improve performance and handle more concurrent requests.
+
+You can control this by adjusting the `WEB_CONCURRENCY` environment variable in the **CONFIG** section of the docker-compose file. [Learn more about the recommended value](./environment-variables.md#web_concurrency) on our ENV variables page.
+
 ### Using an external PostgreSQL server for Mathesar's internal database
 
 If you'd like to use an external PostgreSQL server for Mathesar's internal database, you'll need to do the following:

docs/docs/administration/single-sign-on.md

@@ -50,7 +50,7 @@ Enabling SSO in Mathesar requires setting up a new configuration file or environ
 
 #### 2a. Create configuration file.
 
-You'll either create a file named `sso.yml` or set up the `OIDC_CONFIG_DICT` environmental variable. Which you create, and where you save it depends on how you installed Mathesar:
+You'll either create a file named `sso.yml` or set up the `OIDC_CONFIG_DICT` environment variable. Which you create, and where you save it depends on how you installed Mathesar:
 
 === "For Docker Compose installations"
 
@@ -238,7 +238,7 @@ This means that if you already have users in Mathesar, you can transition to SSO
 
 ## How to deactivate SSO
 
-You can remove the option for users to use SSO by deleting the `sso.yml` file or `OIDC_CONFIG_DICT` environmental variable and restarting Mathesar. Mathesar will then only support password-based authentication.
+You can remove the option for users to use SSO by deleting the `sso.yml` file or `OIDC_CONFIG_DICT` environment variable and restarting Mathesar. Mathesar will then only support password-based authentication.
 
 !!! danger "Deactivating SSO may require manual password resets"
 	Note that Mathesar users who only logged in via SSO will not have known passwords. If SSO is deactivated, these users will not be able to log in until an admin resets their password through the Mathesar UI.

docs/docs/releases/0.2.0.md

@@ -464,8 +464,9 @@ Mathesar 0.2.0 is our first beta release. As such, direct upgrades from our alph
 However, you can migrate the databases created in older versions to work with 0.2.0 by following the steps below.
 
 ### Migration Overview
+
 - **What you can migrate**: the databases you've created/connected and the data within them.
-- **What you cannot migrate**: [user](../user-guide/users.md) accounts, passwords, [explorations](../user-guide/data-explorer.md), and [metadata](../user-guide/metadata.md).
+- **What you cannot migrate**: [user](../user-guide/users.md) accounts, passwords, [explorations](../user-guide/data-explorer.md), and [metadata](../user-guide/databases.md#metadata).
 - Here’s what we’ll do:
     - Take note of the databases you’ve created or connected in your current Mathesar setup.
     - Install Mathesar `0.2.0`.