DRS Write Support

[briandoconnor]

About

DRS write was a feature highlighted during the Plenary in 2024 as well as the April 2025 Connect meeting. See DRS 1.6 priorities whiteboard.

This also came up during the July 2025 eLwazi GA4GH Hackathon. The primary use case is the eLwazi GIF project which looks to have multiple imputation nodes in a federated network. They need to be able to provide back resulting output files and writing these back to a DRS server would be a convenient way to collect results across a federated network:

Having DRS write in 1.6, as an optional extension to DRS, would be really helpful in federated networks like this where analysis happens in a distributed fashion but outputs are consolidated back to a central DRS location.

Feature Branch

See https://github.com/ga4gh/data-repository-service-schemas/tree/feature/issue-415-write-support

For documentation, see docs built from the feature branch: https://ga4gh.github.io/data-repository-service-schemas/preview/feature/issue-415-write-support/docs/

Possible Solutions

All DRS server implementations to date have some out-of-band mechanism to create new DRS entries. To date, there has been no effort to consolidate the mechanism that objects are added to DRS servers. Several approaches could be taken, here's some ideation from Claude Code:

Approach 1: RESTful CRUD Operations

Extend existing endpoints with HTTP methods:

• POST /objects - Create new DRS object
• PUT /objects/{object_id} - Update existing DRS object
• DELETE /objects/{object_id} - Delete DRS object
• PATCH /objects/{object_id} - Partial updates

Pros: Follows REST conventions, intuitive for developers
Cons: May require significant changes to existing client libraries

Approach 2: Separate Write API Namespace

Create dedicated write endpoints:

• POST /write/objects - Create objects
• PUT /write/objects/{object_id} - Update objects
• DELETE /write/objects/{object_id} - Delete objects

Pros: Clear separation between read/write operations, easier to implement optional support
Cons: API surface area expansion, potential confusion about which endpoints to use

Approach 3: Multipart Upload with Metadata

Support data + metadata in single operation:

• POST /objects with multipart form data containing both file content and DRS metadata
• Support for resumable uploads for large files
• Automatic checksum generation and validation

Pros: Atomic operation, handles both data and metadata together
Cons: More complex implementation, may not fit all storage backends

Approach 4: Two-Phase Write Process

Separate metadata creation from data upload:

1. POST /objects - Create DRS object metadata, returns upload URLs
2. Client uploads data to returned URLs (could be cloud storage URLs)
3. POST /objects/{object_id}/finalize - Mark upload complete

Pros: Flexible for different storage backends, supports large file uploads
Cons: More complex client flow, potential for orphaned metadata

Approach 5: Bulk Operations Extension

Extend existing bulk operations pattern:

• POST /bulkobjects/write - Create multiple objects
• Support for batch metadata + file references
• Leverage existing bulk patterns in DRS 1.4+

Pros: Consistent with existing bulk operations, efficient for federated scenarios
Cons: May be overkill for single object operations

Cross-Cutting Considerations

Authorization & Permissions:

• How to handle write permissions differently from read permissions
• Integration with existing passport/authorization mechanisms
• Scope-based access control for write operations

Versioning & Immutability:

• Whether DRS objects should be immutable (create new versions) vs mutable
• How to handle updates to existing objects that may be referenced elsewhere
• Version history and rollback capabilities

Storage Backend Flexibility:

• Support for direct uploads vs proxy uploads
• Cloud storage integration (S3, GCS, Azure presigned URLs)
• On-premise storage considerations

Error Handling & Validation:

• Checksum validation during upload
• Rollback mechanisms for failed operations
• Progress reporting for large uploads

Optional Extension Design:

• How servers advertise write capability support
• Graceful degradation for read-only implementations
• Feature discovery mechanisms

[briandoconnor]

Here's a proposed approach developed with Claude Code:

Enhanced Write Flow Design

1. Service Info Extension - Write Capabilities Discovery

Extended service-info response:

  {
    "id": "com.example.drs",
    "name": "Example DRS",
    "type": {
      "group": "org.ga4gh",
      "artifact": "drs"
    },
    "write_capabilities": {
      "supported": true,
      "storage_locations": [
        {
          "id": "us-east-1-s3",
          "type": "s3",
          "region": "us-east-1",
          "description": "AWS S3 US East 1",
          "public": false
        },
        {
          "id": "europe-gcs",
          "type": "gs",
          "region": "europe-west1",
          "description": "Google Cloud Storage Europe",
          "public": false
        },
        {
          "id": "local-storage",
          "type": "file",
          "description": "On-premise storage",
          "public": true
        }
      ],
      "max_object_size": 5497558138880,
      "supported_checksums": ["md5", "sha256", "crc32c"]
    }
  }

2. Write Authorization Check Endpoint

Check write permissions for authenticated user:
GET /write/authorizations
Authorization: Bearer

Response:

  {
    "authorized_locations": [
      {
        "storage_location_id": "us-east-1-s3",
        "permissions": ["create", "update", "delete"],
        "quota_remaining": 1073741824000
      },
      {
        "storage_location_id": "europe-gcs",
        "permissions": ["create"],
        "quota_remaining": 536870912000
      }
    ]
  }

3. Updated Create Flow with Location Selection

Phase 1: Initialize Object with Target Location
POST /objects
Authorization: Bearer
Content-Type: application/json

  {
    "name": "sample-data.bam",
    "size": 1073741824,
    "checksums": [
      {"type": "md5", "checksum": "d41d8cd98f00b204e9800998ecf8427e"}
    ],
    "target_storage_location": "us-east-1-s3",
    "description": "Sample BAM file from analysis pipeline"
  }

Response: Single Upload URL for Specified Location

  {
    "object": {
      "id": "abc123",
      "name": "sample-data.bam",
      "self_uri": "drs://drs.example.org/abc123",
      "size": 1073741824,
      "created_time": "2025-08-01T10:00:00Z",
      "state": "pending",
      "target_storage_location": "us-east-1-s3",
      "checksums": [...],
      "access_methods": []
    },
    "upload_url": {
      "storage_location_id": "us-east-1-s3",
      "url": "https://s3.amazonaws.com/bucket/abc123?X-Amz-Signature=...",
      "method": "PUT",
      "headers": {"Content-Type": "application/octet-stream"},
      "expires": "2025-08-01T11:00:00Z"
    }
  }

4. Request Additional Upload URLs (Before Finalize)

Get upload URL for additional location:
POST /objects/abc123/upload-urls
Authorization: Bearer

  {
    "storage_location_id": "europe-gcs"
  }

Response:

  {
    "upload_url": {
      "storage_location_id": "europe-gcs",
      "url": "https://storage.googleapis.com/bucket/abc123?GoogleAccessId=...",
      "method": "PUT",
      "headers": {"Content-Type": "application/octet-stream"},
      "expires": "2025-08-01T11:00:00Z"
    }
  }

5. Enhanced Finalize with Multiple Locations

Finalize with locations uploaded to:
POST /objects/abc123/finalize
Authorization: Bearer

  {
    "uploaded_locations": [
      {
        "storage_location_id": "us-east-1-s3",
        "checksums": [{"type": "md5", "checksum": "d41d8cd98f00b204e9800998ecf8427e"}]
      },
      {
        "storage_location_id": "europe-gcs",
        "checksums": [{"type": "md5", "checksum": "d41d8cd98f00b204e9800998ecf8427e"}]
      }
    ]
  }

Response: Available Object with Multiple Access Methods

  {
    "id": "abc123",
    "state": "available",
    "access_methods": [
      {
        "type": "s3",
        "access_url": {"url": "s3://bucket/abc123"},
        "region": "us-east-1"
      },
      {
        "type": "gs",
        "access_url": {"url": "gs://bucket/abc123"},
        "region": "europe-west1"
      }
    ]
  }

New Schema Components

StorageLocation Schema

  StorageLocation:
    type: object
    required: [id, type]
    properties:
      id:
        type: string
        description: Unique identifier for this storage location
      type:
        type: string
        enum: [s3, gs, azure, file, ftp]
        description: Storage backend type
      region:
        type: string
        description: Geographic region or availability zone
      description:
        type: string
        description: Human-readable description
      public:
        type: boolean
        description: Whether this location allows public read access

WriteCapabilities Schema

  WriteCapabilities:
    type: object
    required: [supported]
    properties:
      supported:
        type: boolean
        description: Whether this server supports write operations
      storage_locations:
        type: array
        items:
          $ref: '#/components/schemas/StorageLocation'
      max_object_size:
        type: integer
        format: int64
        description: Maximum size in bytes for uploaded objects
      supported_checksums:
        type: array
        items:
          type: string
        description: Supported checksum algorithms

AuthorizedLocation Schema

  AuthorizedLocation:
    type: object
    required: [storage_location_id, permissions]
    properties:
      storage_location_id:
        type: string
      permissions:
        type: array
        items:
          type: string
          enum: [create, update, delete]
      quota_remaining:
        type: integer
        format: int64
        description: Remaining storage quota in bytes

Key Benefits

1. Multi-Location Support: Users can upload to multiple cloud regions/providers
2. Authorization Aware: Check permissions before attempting uploads
3. Efficient URL Management: Request upload URLs on-demand to avoid expiration
4. Flexible Replication: Upload to additional locations before finalizing
5. Resource Management: Quota tracking per location
6. Discovery: Service capabilities clearly advertised

Workflow Examples

Simple single-location upload:

1. Check service-info → see available locations
2. POST /objects with target_storage_location
3. Upload to provided URL
4. POST /objects/{id}/finalize

Multi-location replication:

1. Check write authorizations → see permitted locations
2. POST /objects with primary location
3. Upload to primary location
4. POST /objects/{id}/upload-urls for secondary location
5. Upload to secondary location
6. POST /objects/{id}/finalize with both locations

[briandoconnor]

Changes produced by Claude Code

Complete Implementation Summary

🎯 Core Features Implemented:

1. RESTful CRUD Operations (Approach 1)
   - POST /objects - Create new DRS objects
   - PUT /objects/{id} - Update existing objects
   - DELETE /objects/{id} - Delete objects
2. Two-Phase Upload Process (Approach 4)
   - Phase 1: Initialize object with metadata → receive upload URL
   - Phase 2: Upload to cloud storage → finalize object
   - POST /objects/{id}/finalize - Complete the upload process
3. Multi-Location Storage Support
   - Storage location discovery via service-info
   - Authorization checking per location
   - Request additional upload URLs for multiple locations
   - Support for S3, GCS, Azure, and other storage backends
4. Authorization & Discovery
   - GET /write/authorizations - Check user permissions
   - Extended service-info with write capabilities
   - Per-location quota and permission management

📊 New Schemas Created:

• StorageLocation - Define storage backends
• WriteCapabilities - Advertise server write support
• UploadURL - Presigned upload URLs with metadata
• AuthorizedLocation - User permissions per location
• CreateObjectRequest/Response - Object creation flow
• UpdateObjectRequest - Object update operations
• FinalizeObjectRequest - Complete upload process

🛠 New API Endpoints:

• POST /objects - Create objects
• PUT/DELETE /objects/{id} - Update/delete objects
• POST /objects/{id}/upload-urls - Request additional upload URLs
• POST /objects/{id}/finalize - Complete uploads
• GET /write/authorizations - Check write permissions

📋 Enhanced Error Handling:

• 409 Conflict - Object state issues
• 422 Unprocessable Entity - Checksum validation failures
• Enhanced 400 Bad Request examples
• Object state management (pending → available)

📖 Documentation:

• Comprehensive WriteOperations.md guide
• Multi-location upload examples
• Best practices and error handling guidance

[briandoconnor]

I opened a PR with a possible solution for this issue. Let's discuss there: #416