feat: add middleware management API specs

[keshxvdayal]

Detailed Implementation Plan

1. Create OpenAPI Specification Document

   • Create data/specs/middleware_management.yaml
   • Define base paths and versioning (/ga4gh/tes/v1/middlewares)
   • Document all endpoints:
     • GET /middlewares - List all configured middlewares
     • POST /middlewares - Add new middleware (local or from GitHub)
     • GET /middlewares/{middleware_id} - Get middleware details
     • PUT /middlewares/{middleware_id} - Update middleware configuration
     • DELETE /middlewares/{middleware_id} - Remove middleware
     • PUT /middlewares/reorder - Reorder middleware stack
     • POST /middlewares/validate - Validate middleware code without adding
   • Define comprehensive schemas:
     • MiddlewareConfig - Full middleware configuration object
     • MiddlewareCreate - Request body for creating middleware
     • MiddlewareUpdate - Request body for updating middleware
     • MiddlewareList - Array response with pagination support
     • MiddlewareOrder - Order configuration for reordering
     • ValidationRequest - Code validation request
     • ErrorResponse - Standard error format
   • Document query parameters (pagination, filtering, sorting)
   • Define all possible HTTP status codes and error responses

2. Integrate with FOCA Configuration

   • Update pro_tes/config.yaml to reference the new spec
   • Add middleware API paths to FOCA routing configuration
   • Configure CORS and authentication settings (if applicable)
   • Set up request/response validation middleware via FOCA

3. Write API Documentation

   • Create docs/api/middleware_management.md
   • Provide usage examples with curl commands
   • Document authentication requirements
   • Explain order/priority system for middleware execution
   • Document fallback group behavior
   • Add sequence diagrams for complex operations (reordering, fallback)

4. Validation and Review

   • Validate OpenAPI spec with openapi-generator validate
   • Generate API documentation preview with Redoc/Swagger UI
   • Create examples for each endpoint operation
   • Peer review with team focusing on:
     • API design best practices
     • Consistency with existing proTES APIs
     • GA4GH TES specification alignment

Summary by Sourcery

Add an OpenAPI specification and configuration wiring for a new runtime middleware management API in proTES.

New Features:

• Introduce a Middleware Management REST API specification with endpoints for listing, creating, updating, deleting, reordering, and validating middlewares.

Enhancements:

• Extend FOCA configuration to load the new middleware management OpenAPI spec with strict request/response validation and routing to middleware controllers.
• Add a MongoDB index on middleware name to support efficient lookup and management operations.

Documentation:

• Add middleware management architecture and API design documentation describing endpoints, data models, design decisions, and future work.

[sourcery-ai]

Reviewer's Guide

Adds a new OpenAPI 3.0 specification for runtime middleware management and wires it into FOCA/Connexion configuration and project docs, without yet implementing controller logic.

Sequence diagram for adding a new middleware via the OpenAPI spec

sequenceDiagram
  actor Admin
  participant Client as API_client
  participant Connexion as FOCA_Connexion
  participant Controller as MiddlewaresController
  participant DB as MongoDB_middlewares

  Admin->>Client: Prepare MiddlewareCreate_payload
  Client->>Connexion: POST /ga4gh/tes/v1/middlewares
  Connexion->>Connexion: Validate_request_against_OpenAPI
  Connexion->>Controller: addMiddleware(MiddlewareCreate)

  Controller->>DB: Query existing_middlewares_for_order
  DB-->>Controller: Existing_middlewares_with_orders
  Controller->>Controller: Compute_final_order_and_shift_stack
  Controller->>DB: Insert new_middleware_document
  DB-->>Controller: Insert_result_with_id

  Controller-->>Connexion: 201 MiddlewareCreateResponse
  Connexion-->>Client: HTTP_201_with_JSON_body
  Client-->>Admin: Show_created_middleware_id_and_order

Loading

Class diagram for middleware management OpenAPI schemas

classDiagram
  class MiddlewareConfig {
    string _id
    string name
    string class_path_string
    string[] class_path_group
    int order
    string source
    string github_url
    object config
    boolean enabled
    string created_at
    string updated_at
  }

  class MiddlewareCreate {
    string name
    string class_path_string
    string[] class_path_group
    int order
    string github_url
    object config
    boolean enabled
  }

  class MiddlewareUpdate {
    string name
    int order
    object config
    boolean enabled
  }

  class MiddlewareList {
    MiddlewareConfig[] middlewares
    int total
    int limit
    int offset
  }

  class MiddlewareCreateResponse {
    string _id
    int order
    string message
  }

  class MiddlewareOrder {
    string[] ordered_ids
  }

  class ValidationRequest {
    string class_path
    string github_url
    string code
  }

  class ValidationErrorItem {
    int line
    int column
    string message
    string severity
  }

  class ValidationWarningItem {
    int line
    string message
    string severity
  }

  class ValidationResponse {
    boolean valid
    string message
    ValidationErrorItem[] errors
    ValidationWarningItem[] warnings
  }

  class ErrorResponse {
    string error
    string message
    object details
    string timestamp
  }

  MiddlewareList "1" --> "*" MiddlewareConfig : contains
  MiddlewareCreateResponse --> MiddlewareConfig : _id_and_order_align_with
  MiddlewareOrder "1" --> "*" MiddlewareConfig : orders
  ValidationResponse "1" --> "*" ValidationErrorItem : has
  ValidationResponse "1" --> "*" ValidationWarningItem : has

Loading

File-Level Changes

Change	Details	Files
Integrate middleware management OpenAPI spec into FOCA API configuration and database indexes.	Add MongoDB index configuration for middlewares collection keyed by name. Register new middleware_management OpenAPI spec in FOCA specs list with strict request/response validation enabled. Configure routing to pro_tes.api.middlewares.controllers and temporarily disable authentication for these endpoints. Tidy custom middleware list config (no behavioral change).	pro_tes/config.yaml
Introduce Middleware Management OpenAPI 3.0 specification describing middleware lifecycle operations and data models.	Define seven middleware management endpoints for listing, creating, retrieving, updating, deleting, reordering, and validating middlewares under /ga4gh/tes/v1/middlewares. Add query parameters for pagination, sorting, and filtering on list endpoint and force flag on delete. Specify request/response schemas for middleware CRUD, ordering, validation, and error handling, including MongoDB ObjectId patterns and GitHub URL constraints. Model middleware configuration with support for single class paths and fallback groups, tracking source (local/github), timestamps, and soft-delete semantics.	pro_tes/api/middleware_management.yaml
Add high-level documentation for the Middleware Management API design and its role in the project.	Document the purpose and background of the middleware management feature and its design-first approach. Summarize the defined endpoints, schemas, and key design decisions (ordering, soft delete, immutable class paths, GitHub integration). Describe integration with FOCA/Connexion, file layout, validation strategy, and planned future subtasks for implementation and security. Clarify that this PR is non-breaking and currently limited to specification and documentation work.	docs/middleware.md

Possibly linked issues

• #N/A: PR creates the full middleware_management OpenAPI spec, schemas, endpoints, and docs exactly as the issue requests.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 2 issues, and left some high level feedback:

• The MiddlewareCreateResponse schema only exposes _id, order, and message, but the docs in docs/middleware.md state that it returns the full middleware object; consider either expanding the response schema (e.g., embed MiddlewareConfig) or updating the docs to reflect the actual payload to avoid confusion for API consumers.
• In ValidationRequest, the class_path field is described as "Class path or code to validate" while a separate code field is also defined; it would be clearer to tighten the descriptions and possibly add explicit rules (e.g., mutual exclusivity or precedence) so clients know exactly how these fields should be used together.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The `MiddlewareCreateResponse` schema only exposes `_id`, `order`, and `message`, but the docs in `docs/middleware.md` state that it returns the full middleware object; consider either expanding the response schema (e.g., embed `MiddlewareConfig`) or updating the docs to reflect the actual payload to avoid confusion for API consumers.
- In `ValidationRequest`, the `class_path` field is described as "Class path or code to validate" while a separate `code` field is also defined; it would be clearer to tighten the descriptions and possibly add explicit rules (e.g., mutual exclusivity or precedence) so clients know exactly how these fields should be used together.

## Individual Comments

### Comment 1
<location> `pro_tes/config.yaml:82` </location>
<code_context>
+      - api/middleware_management.yaml
+    add_operation_fields:
+      x-openapi-router-controller: pro_tes.api.middlewares.controllers
+    disable_auth: True
+    connexion:
+      strict_validation: True
</code_context>

<issue_to_address>
**🚨 issue (security):** Exposing middleware management without auth looks risky for production use.

Since this endpoint can change the middleware stack at runtime, keeping `disable_auth: True` lets anyone with network access reorder/enable/disable/add middlewares. Unless this is strictly limited to a trusted/debug environment, this should require authz (or be configurable per deployment) to avoid a straightforward privilege escalation path.
</issue_to_address>

### Comment 2
<location> `pro_tes/api/middleware_management.yaml:572-358` </location>
<code_context>
+            - "507f1f77bcf86cd799439012"
+            - "507f1f77bcf86cd799439013"
+
+    ValidationRequest:
+      type: object
+      description: Request body for validating middleware code
+      required:
+        - class_path
+      properties:
</code_context>

<issue_to_address>
**issue (bug_risk):** ValidationRequest requires class_path even when validating raw code, which conflicts with the description.

The schema contradicts the docs: `code` is documented as an alternative to `class_path`, but `class_path` is required. This blocks use cases where clients submit only raw code or only a `github_url`. Please adjust the schema (e.g., with `oneOf`/`anyOf` to require at least one of `class_path`, `github_url`, or `code`, or by making `class_path` optional and clarifying the validation rules) so the spec matches the intended behavior.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[Copilot]

Pull request overview

This PR adds a comprehensive OpenAPI 3.0 specification for middleware management functionality in proTES, enabling dynamic management of middleware components at runtime. This is the first subtask in a multi-part implementation following a design-first approach, focusing solely on the API specification and documentation without implementing the actual controller logic.

Changes:

• Added complete OpenAPI 3.0 specification defining 7 REST endpoints for middleware CRUD operations, reordering, and validation
• Configured FOCA integration with new middleware management API routes and MongoDB collection
• Added comprehensive documentation describing the API design, endpoints, data models, and implementation plan

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 17 comments.

File	Description
pro_tes/api/middleware_management.yaml	New OpenAPI 3.0 specification defining middleware management API with 7 endpoints and 9 schemas
pro_tes/config.yaml	Added FOCA integration for middleware API, database collection configuration, and routing setup
docs/middleware.md	Comprehensive documentation covering API design, implementation details, security considerations, and future work

Comments suppressed due to low confidence (1)

docs/middleware.md:210

• The changelog date "2026-01-24" is in the future. This should be corrected to use the actual date when this work was completed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[uniqueg]

I didn't have a chance to review everything yet, but please go ahead and address the existing comments.

[uniqueg]

Still some ways to go, but we're getting there! :)

[uniqueg]

Okay, this is starting to look much better now. There are four issues left, 2 previously existing ones, and 2 new ones. With those addressed, I think we are good to go.

Previously existing issues/comments:

• Completely remove enabled from all models, examples & descriptions etc.; we don't want to disable/enable middlewares for now
• Ensure there is a way how middlewares within a fallback group can be reordered and deleted

New issues/comments:

• The MiddlewareCreate model is redundant (using readOnly was also addressed previously, but ignored)
• MiddlewareSource needs some improvements

After you have addressed an issue, please leave a message in the conversation where you briefly confirm what you did. This makes it much easier for reviewing. But don't resolve the issues, because that risks that issues are overlooked - leave resolving issues up to the people who have raised them.

[uniqueg]

This looks good, thanks a lot. Please address the other 4 comments I have sent in the chat. Then we're ready to merge.

[uniqueg]

Great job, thanks a lot, @keshxvdayal