This document describes the implementation of Subtask 1 for the Middleware Management feature in proTES. This subtask focuses on designing and documenting the API specification that will enable dynamic middleware management at runtime.
The proTES project required a way to manage middleware components dynamically without restarting the service. The maintainer requested that this feature be broken down into smaller, independently mergeable pull requests following a design-first approach. This subtask represents the foundation: the OpenAPI specification that defines how the API will behave.
This subtask delivers a complete OpenAPI 3.0 specification for middleware management. The specification defines seven REST endpoints that cover all necessary operations for middleware lifecycle management.
List Middlewares - GET /ga4gh/tes/v1/middlewares Returns all configured middlewares with pagination and filtering support. Results are sorted by execution order by default. Supports filtering by enabled status and source type.
Add Middleware - POST /ga4gh/tes/v1/middlewares Creates a new middleware in the execution stack. Supports loading from local class paths or GitHub repositories. Automatically handles order assignment and stack shifting.
Get Middleware Details - GET /ga4gh/tes/v1/middlewares/{middleware_id} Retrieves detailed information about a specific middleware including configuration, metadata, and execution statistics.
Update Middleware - PUT /ga4gh/tes/v1/middlewares/{middleware_id} Updates middleware configuration. Only allows modification of name, order, config parameters, and enabled status. Class path cannot be changed for security reasons.
Delete Middleware - DELETE /ga4gh/tes/v1/middlewares/{middleware_id} Removes a middleware from the stack. Supports soft delete (disable) by default and hard delete with force parameter.
Reorder Stack - PUT /ga4gh/tes/v1/middlewares/reorder Reorders the entire middleware execution stack by accepting an ordered array of middleware IDs.
Validate Code - POST /ga4gh/tes/v1/middlewares/validate Validates middleware code before creation. Checks Python syntax, required interface implementation, and security constraints.
The API uses nine schema definitions to structure request and response data:
MiddlewareConfig: Complete middleware representation including ID, name, class path, execution order, enabled status, configuration parameters, source information, and timestamps.
MiddlewareCreate: Request body for creating new middleware. Includes name, class path (string or array for fallback groups), optional order, enabled flag, configuration dict, and optional GitHub URL.
MiddlewareUpdate: Request body for updates. Limited to name, order, config, and enabled fields to prevent unauthorized code changes.
MiddlewareList: Paginated list response containing middleware array and total count.
MiddlewareCreateResponse: Response after successful creation including the new middleware object and a success message.
MiddlewareOrder: Request body for reordering containing an array of middleware IDs in desired execution order.
ValidationRequest: Code validation request containing Python code string to validate.
ValidationResponse: Validation result including validity boolean, validation messages array, detected class name, and required methods check.
ErrorResponse: Standard error response with status code, error type, and detailed message.
MongoDB ObjectId Format: Uses 24-character hexadecimal strings for middleware identification. This aligns with the existing proTES database schema and provides guaranteed uniqueness.
Order-Based Execution: Middlewares execute in ascending order. Lower order values run first. This provides clear, predictable execution flow that's easy to understand and debug.
Fallback Group Support: Allows multiple class paths in a single middleware entry. If the first middleware fails, the system automatically tries the next one in the list. This improves reliability without complex error handling.
Soft Delete Default: DELETE operations disable rather than remove middlewares by default. This preserves execution history and allows easy rollback. Hard delete requires explicit force parameter.
Immutable Class Path: Once created, a middleware's class path cannot be changed. This prevents security risks from code substitution attacks. To change implementation, users must delete and recreate.
GitHub Integration: Supports loading middleware code directly from GitHub URLs. The system fetches, validates, and caches the code. This enables sharing middleware across deployments without manual file management.
Source Tracking: Records whether middleware originated from local files or GitHub. Helps administrators understand deployment composition and troubleshoot issues.
Validation Endpoint: Separate endpoint for validating middleware code before creation. Prevents deployment of broken middleware and provides immediate feedback on implementation issues.
The specification integrates with proTES's existing FOCA configuration framework. Added configuration block:
specs:
- path:
- api/middleware_management.yaml
add_operation_fields:
x-openapi-router-controller: pro_tes.api.middlewares.controllers
disable_auth: True
connexion:
strict_validation: True
validate_responses: TrueThis configuration tells FOCA to:
- Load the OpenAPI spec from the api directory
- Route requests to the middlewares controller module
- Disable authentication for initial development (will be secured in Subtask 4)
- Enable strict validation of requests and responses
The FOCA framework uses Connexion under the hood, which automatically generates routing, parameter validation, and response serialization based on the OpenAPI specification.
pro_tes/
├── api/
│ └── middleware_management.yaml (OpenAPI specification)
└── config.yaml (FOCA integration)
docs/
├── api/
│ ├── middleware_management.md (API documentation)
│ ├── middleware_management.postman_collection.json
│ └── QUICK_REFERENCE.md
├── architecture/
│ └── middleware_api_design.md (Architecture decisions)
└── middleware.md (This file)
scripts/
└── validate_openapi.sh (Validation utility)
API Documentation: Comprehensive guide with request/response examples for each endpoint. Includes curl commands, common use cases, and troubleshooting tips.
Architecture Decision Record: Documents twelve major design decisions with rationale, alternatives considered, and consequences. Serves as reference for future development.
Postman Collection: Ready-to-use collection with fourteen pre-configured requests. Includes environment variables, test scripts, and example data for all scenarios.
Quick Reference: Single-page reference with essential endpoints, parameters, and response codes. Designed for daily development use.
Validation Script: Bash script that validates OpenAPI syntax using multiple tools. Checks for common errors like undefined schema references and invalid endpoint definitions.
This subtask focuses on specification validation rather than runtime testing since no executable code is implemented yet. Validation performed:
YAML Syntax: Verified file parses correctly as valid YAML without syntax errors.
OpenAPI Compliance: Confirmed specification follows OpenAPI 3.0 standards including required fields, valid schema definitions, and proper reference resolution.
Schema Completeness: Validated all endpoints reference defined schemas and all schemas include required properties with appropriate types.
Path Coverage: Verified all seven endpoints are defined with appropriate HTTP methods and parameters.
Runtime testing will occur in Subtask 2 when controllers are implemented.
While authentication is disabled for initial development, the specification includes security design:
Input Validation: All parameters include type, format, and constraint definitions. Connexion will automatically validate inputs before they reach controller code.
MongoDB ObjectId Pattern: Enforces 24-character hex pattern preventing injection attacks through malformed IDs.
Class Path Immutability: Prevents code substitution attacks by making class paths unchangeable after creation.
Code Validation: Separate validation endpoint allows testing code safety before deployment.
Source Tracking: Records code origin for audit and security review purposes.
Full security implementation including authentication, authorization, and rate limiting will be added in Subtask 4.
This subtask completes the API design phase. Subsequent subtasks will build on this foundation:
Subtask 2: Implement controller logic to handle API requests and interact with MongoDB.
Subtask 3: Build dynamic middleware loading system that instantiates classes and manages execution stack at runtime.
Subtask 4: Add authentication, authorization, RBAC controls, and API security.
Subtask 5: Implement monitoring, logging, and metrics collection for middleware operations.
Subtask 6: Complete integration testing, update deployment configurations, and finalize documentation.
External:
- OpenAPI 3.0 specification format
- FOCA framework (Flask-based configuration)
- Connexion (OpenAPI request routing)
- MongoDB (persistence layer)
Internal:
- Existing proTES API structure
- Current middleware plugin architecture
- MongoDB database configuration
None. This subtask only adds new API endpoints without modifying existing functionality.
Specification validated successfully:
- 7 API endpoints defined
- 9 schema definitions complete
- All references resolve correctly
- YAML syntax valid
- OpenAPI 3.0 compliance confirmed Location: pro_tes/api/middleware_management.yaml
2026-01-24: Initial OpenAPI specification completed
- Defined 7 REST endpoints for middleware management
- Created 9 schema definitions
- Integrated with FOCA configuration
- Delivered comprehensive documentation suite
- Validated specification structure and syntax