Added an S3 endpoint and a duplicate-file validation with feedback in frontend

Author: naquib314

README.md

@@ -10,37 +10,111 @@ See deployment for notes on how to deploy the project on a live system (Coming s
 
 ## Prerequisites
 
-Taiga 2 is built on the following stack:
+- **Python 3.10** (required — the project pins `>=3.10,<3.11`)
+- **Poetry** (dependency manager)
+- **Node.js** and **Yarn** (for the React frontend)
+- **Redis** (Celery broker and result backend)
+- **Docker** (optional — only needed if you want to test file uploads via MiniStack)
 
-### Database
+### Stack overview
 
-- PostgreSQL (to store the app information)
-- Psycopg2
-- SQLAlchemy
-- Amazon S3 (to store the files)
+| Layer | Technologies |
+|-------|-------------|
+| Database | SQLite (local dev), PostgreSQL (production), SQLAlchemy |
+| Backend/API | Python 3.10, Flask, Connexion (Swagger/OpenAPI), Celery/Redis |
+| Object storage | AWS S3 (production), MiniStack (optional, local dev) |
+| Frontend | React, TypeScript, Webpack, Yarn |
 
-### Backend/API
+## Installing
+
+1. Install Python dependencies:
+
+        poetry install
+
+2. Install frontend dependencies:
+
+        cd react_frontend && yarn install && cd ..
+
+3. Copy the sample settings file (if you don't already have one):
+
+        cp settings.cfg.sample settings.cfg
+
+4. Create the dev database:
+
+        poetry run bash -c 'source setup_env.sh && flask recreate-dev-db'
+
+## Running Locally
+
+Start these four processes in separate terminal windows:
+
+```bash
+# Terminal 1 — Redis (skip if already running; check with `redis-cli ping`)
+redis-server
+
+# Terminal 2 — Webpack dev server (frontend hot reload)
+poetry run bash -c 'source setup_env.sh && flask webpack'
+
+# Terminal 3 — Flask app server
+poetry run bash -c 'source setup_env.sh && flask run'
+
+# Terminal 4 — Celery worker (async file conversion tasks)
+poetry run bash -c 'source setup_env.sh && flask run-worker'
+```
+
+Open your browser to: **http://127.0.0.1:5000/taiga/**
+
+You are automatically logged in as the seeded admin user (`admin@broadinstitute.org`) via the `DEFAULT_USER_EMAIL` setting.
+
+Without S3 configured, you can browse/search the seeded data, create folders, and work with the UI. File uploads require either MiniStack or real AWS credentials (see below).
+
+## Local S3 with MiniStack (Optional)
+
+[MiniStack](https://github.com/Nahuel990/ministack) is a free, open-source AWS emulator that runs 33 AWS services (including S3 and STS) in a single Docker container. It lets you test the full upload pipeline locally without an AWS account.
+
+### Setup
+
+1. Start MiniStack:
+
+        docker run -d --name ministack -p 4566:4566 nahuelnucera/ministack
+
+2. Create the local S3 bucket (run once):
+
+    ```bash
+    python -c "import boto3; boto3.client('s3', endpoint_url='http://localhost:4566', aws_access_key_id='test', aws_secret_access_key='test').create_bucket(Bucket='taiga-dev')"
+    ```
 
-- Python 3.6
-- Flask
-- Celery/Redis
-- Swagger
+3. In `settings.cfg`, uncomment the MiniStack block (Option A) and comment out Option B:
 
-### FrontEnd
+    ```python
+    S3_ENDPOINT_URL = 'http://localhost:4566'
+    AWS_ACCESS_KEY_ID = 'test'
+    AWS_SECRET_ACCESS_KEY = 'test'
+    S3_BUCKET = 'taiga-dev'
+    ```
 
-- React
-- TypeScript
-- Webpack
-- Yarn
+4. Restart Flask and the Celery worker to pick up the new settings.
 
-### Configuring AWS users
+### Managing MiniStack
+
+```bash
+docker start ministack   # start (if previously stopped)
+docker stop ministack    # stop
+docker rm ministack      # remove entirely
+```
+
+### Switching back to no-S3 mode
+
+Set `S3_ENDPOINT_URL = ''` and clear the AWS keys in `settings.cfg`. The app runs fine without S3 — you just can't upload files.
+
+## Configuring AWS (Production)
 
 We need two users: One IAM account (main) is used in general by the app to read/write to S3. The second (uploader) has it's rights delegated via STS on a short term basis. However, this user should
 only have access to upload to a single location within S3.
 
 Permissions for the main user:
 
-```{
+```json
+{
     "Version": "2012-10-17",
     "Statement": [
         {
@@ -57,7 +131,8 @@ Permissions for the main user:
 
 Permissions for the "upload" user:
 
-```{
+```json
+{
     "Version": "2012-10-17",
     "Statement": [
         {
@@ -112,46 +187,13 @@ For our case, it is pretty simple:
 
 #### Configure Taiga to use your Bucket
 
-1. Copy `settings.cfg.sample` to `settings.cfg`
-2. edit `settings.cfg` and set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
-3. also set S3_BUCKET to the bucket created above
-
-## Installing
-
-1.  Install all the dependencies:
-
-        `poetry install`
-        `poetry shell`
-
-2.  Create a test database to have some data to work with:
-
-        `./flask recreate-dev-db`
+1. Edit `settings.cfg` and set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
+2. Set `S3_BUCKET` to the bucket created above
+3. Remove `S3_ENDPOINT_URL` (or leave it empty) so the app connects to real AWS
 
-3.  Open 4 terminal windows to launch Webpack, Taiga 2, Celery and Redis processes:
+## Adding user to admin group
 
-    a. In terminal 1:
-
-        `./flask webpack`
-
-    b. In terminal 2:
-
-        `redis-server`
-
-    c. In terminal 3:
-
-        `./flask run`
-
-    d. In terminal 4:
-
-        `./flask run-worker`
-
-4.  Congratulations! You can now access to Taiga 2 through your browser at:
-
-        `http://127.0.0.1:5000/taiga/`
-
-## adding user to admin group
-
-```
+```sql
 INSERT INTO group_user_association (group_id, user_id) select 1, id FROM users WHERE name =
 'pmontgom';
 ```

react_frontend/src/components/UploadTracker.tsx

@@ -207,14 +207,19 @@ export class UploadTracker {
   ): Promise<DatasetIdAndVersionId> {
     // TODO: If we change the page, we lose the download
     // Configure the AWS S3 object with the received credentials
-    let s3 = new AWS.S3({
+    let s3Options: AWS.S3.ClientConfiguration = {
       apiVersion: "2006-03-01",
       credentials: {
         accessKeyId: s3_credentials.accessKeyId,
         secretAccessKey: s3_credentials.secretAccessKey,
         sessionToken: s3_credentials.sessionToken,
       },
-    });
+    };
+    if (s3_credentials.endpointUrl) {
+      s3Options.endpoint = s3_credentials.endpointUrl;
+      s3Options.s3ForcePathStyle = true;
+    }
+    let s3 = new AWS.S3(s3Options);
 
     // Looping through all the files
     let uploadPromises: Array<Promise<boolean>> = [];

react_frontend/src/components/modals/UploadDialogs/UploadDialog.tsx

@@ -123,6 +123,20 @@ export class UploadDialog extends React.Component<
       validationError = "At least one file is required";
     }
 
+    if (!validationError) {
+      const nameCounts = new Map<string, number>();
+      for (const file of this.state.uploadFiles) {
+        const trimmed = file.name.trim();
+        nameCounts.set(trimmed, (nameCounts.get(trimmed) || 0) + 1);
+      }
+      const duplicates = Array.from(nameCounts.entries())
+        .filter(([_, count]) => count > 1)
+        .map(([name]) => name);
+      if (duplicates.length > 0) {
+        validationError = `Duplicate file name${duplicates.length > 1 ? "s" : ""}: ${duplicates.join(", ")}`;
+      }
+    }
+
     let submitButton: any;
 
     // If we have a new datasetVersion in the state, we can show the link button

react_frontend/src/models/models.ts

@@ -236,6 +236,7 @@ export interface S3Credentials {
   sessionToken: string;
   bucket: string;
   prefix: string;
+  endpointUrl?: string;
 }
 
 export interface S3UploadedData {

settings.cfg.sample

@@ -9,14 +9,28 @@ PREFIX = '/taiga'
 USE_FRONTEND_DEV_SERVER = True
 
 # Amazon Web Services
+# Option A — Local dev with MiniStack (no real AWS needed):
+#   docker run -d --name ministack -p 4566:4566 nahuelnucera/ministack
+#   Then uncomment the MiniStack block below and comment out Option B.
+#
+# Option B — Real AWS or no S3 (browse seeded data only):
+#   Leave S3_ENDPOINT_URL empty and fill in real AWS keys, or leave all blank.
+
+# --- Option A: MiniStack ---
+# S3_ENDPOINT_URL = 'http://localhost:4566'
+# AWS_ACCESS_KEY_ID = 'test'
+# AWS_SECRET_ACCESS_KEY = 'test'
+# S3_BUCKET = 'taiga-dev'
+
+# --- Option B: Real AWS (or blank for no uploads) ---
+S3_ENDPOINT_URL = ''
 AWS_ACCESS_KEY_ID = ''
 AWS_SECRET_ACCESS_KEY = ''
+S3_BUCKET = ''
 
 CLIENT_UPLOAD_AWS_ACCESS_KEY_ID = AWS_ACCESS_KEY_ID
 CLIENT_UPLOAD_AWS_SECRET_ACCESS_KEY = AWS_SECRET_ACCESS_KEY
 
-S3_BUCKET = ''
-
 # Database configuration
 SQLALCHEMY_DATABASE_URI = 'sqlite:///db.sqlite3'
 

taiga2/api_app.py

@@ -53,7 +53,7 @@ def create_app(settings_override=None, settings_file=None):
     init_backend_auth(app)
 
     # Exception report with StackDriver
-    exception_reporter.init_app(app=app, service_name="taiga-" + app.config["ENV"])
+    exception_reporter.init_app(app=app, service_name="taiga-" + app.config.get("ENV", "dev"))
     register_errorhandlers(app=app)
 
     return api_app, app

taiga2/controllers/endpoint.py

@@ -213,6 +213,10 @@ def get_s3_credentials():
         "prefix": prefix,
     }
 
+    endpoint_url = flask.current_app.config.get("S3_ENDPOINT_URL")
+    if endpoint_url:
+        model_frontend_credentials["endpointUrl"] = endpoint_url
+
     # See frontend/models/models.ts for the S3Credentials object and Swagger.yaml
     return flask.jsonify(model_frontend_credentials)
 
@@ -637,6 +641,10 @@ def create_dataset(sessionDatasetInfo):
     dataset_description = sessionDatasetInfo.get("datasetDescription", None)
     current_folder_id = sessionDatasetInfo["currentFolderId"]
 
+    duplicates = models_controller.validate_session_has_no_duplicate_filenames(session_id)
+    if duplicates:
+        api_error("Duplicate file names in upload: {}".format(", ".join(duplicates)))
+
     added_dataset = models_controller.add_dataset_from_session(
         session_id, dataset_name, dataset_description, current_folder_id
     )
@@ -723,6 +731,10 @@ def create_new_dataset_version(datasetVersionMetadata):
     dataset_version = datasetVersionMetadata.get("datasetVersion", None)
     current_user = models_controller.get_current_session_user()
 
+    duplicates = models_controller.validate_session_has_no_duplicate_filenames(session_id)
+    if duplicates:
+        api_error("Duplicate file names in upload: {}".format(", ".join(duplicates)))
+
     models_controller.lock()
 
     new_dataset_version = _create_new_dataset_version_from_session(

taiga2/controllers/models_controller.py

@@ -1507,6 +1507,17 @@ def log_datafile_read_access_info(datafile_id: str):
     db.session.commit()
 
 
+def validate_session_has_no_duplicate_filenames(session_id: str):
+    """Check that no two files in the upload session share the same name.
+    Returns a list of duplicate names (empty if none)."""
+    files = get_upload_session_files_from_session(session_id)
+    seen = {}
+    for f in files:
+        name = f.filename.strip()
+        seen[name] = seen.get(name, 0) + 1
+    return [name for name, count in seen.items() if count > 1]
+
+
 def _add_datafiles_from_session(session_id: str):
     # We retrieve all the upload_session_files related to the UploadSession
     added_files = get_upload_session_files_from_session(session_id)

taiga2/default_settings.py

@@ -16,3 +16,6 @@
 
 # S3 settings
 CLIENT_UPLOAD_TOKEN_EXPIRY = 86400  # A day
+
+ENV = "dev"
+REPORT_EXCEPTIONS = False

taiga2/swagger/swagger.yaml

@@ -243,6 +243,8 @@ paths:
                 type: string
               prefix:
                 type: string
+              endpointUrl:
+                type: string
 
   /upload_session:
     get: