Mia-Platform CRUD Service Universal (experimental)

⚠️ EXPERIMENTAL - NOT FOR PRODUCTION USE

This service is in experimental phase and is not ready for production environments. For production deployments, please use the CRUD Service instead.

A high-performance CRUD service written in Rust for both SQL and NoSQL DBs

Supported Databases

The CRUD Service Universal supports a wide range of database types through feature flags:

NoSQL Databases

• MongoDB - Document database (default, always enabled)
• Amazon DocumentDB - MongoDB-compatible document database
• Redis - Key-value store (feature: redis)
• Amazon DynamoDB - Serverless NoSQL database (feature: dynamodb)
• Elasticsearch - Search and analytics engine (feature: elasticsearch)
• Azure Cosmos DB - Multi-model database service (feature: cosmosdb)

SQL Databases

• PostgreSQL - Advanced open-source relational database (feature: sql)
• MySQL - Popular open-source relational database (feature: sql)
• MariaDB - MySQL-compatible database (feature: sql)
• SQLite - Lightweight file-based database (feature: sql)
• Oracle Database - Enterprise relational database (feature: oracle)

Enterprise Databases (via ODBC)

• Microsoft SQL Server - Microsoft's relational database (feature: odbc)
• IBM DB2 - IBM's relational database (feature: odbc)
• SAP HANA - In-memory database platform (feature: odbc)
• Teradata - Analytics database platform (feature: odbc)
• Snowflake - Cloud data warehouse (feature: odbc)
• Generic ODBC - Any ODBC-compatible database (feature: odbc)

Cloud Services

• Google BigTable - Wide-column NoSQL database (feature: bigtable)
• Amazon RDS - Managed relational database service

Note: To enable specific database support, compile with the appropriate feature flags:

# Enable all databases
cargo build --features all-databases

# Enable specific databases
cargo build --features "sql,redis,dynamodb"

Features

• Full CRUD Operations: Create, Read, Update, Delete operations on MongoDB collections
• JSON Schema Validation: Comprehensive validation using JSON Schema
• Document State Management: Support for PUBLIC, DRAFT, TRASH, DELETED states
• Import/Export: Support for JSON, CSV, NDJSON, and Excel formats
• MongoDB Views: Aggregation pipeline support with writable views
• Authentication & Authorization: JWT and ACL support
• Health Checks: Comprehensive health and readiness endpoints
• Metrics: Prometheus metrics integration
• High Performance: Built with Rust and Axum for maximum throughput
• OpenAPI Documentation: Auto-generated Swagger UI

Quick Start

Prerequisites

• Rust 1.75 or later
• MongoDB 4.4 or later
• (Optional) Docker and Docker Compose

Local Development

1. Clone and Setup

   git clone <repository-url>
   cd crud-service-rust
   cp .env.example .env

2. Configure Environment Edit .env file with your MongoDB connection and settings:

   MONGODB_URL=mongodb://localhost:27017/crud_service_rust
   COLLECTION_DEFINITION_FOLDER=./examples/collections
   USER_ID_HEADER_KEY=userid

3. Start MongoDB

   # Using Docker
   docker run --name mongodb -p 27017:27017 -d mongo:7.0
   
   # Or use existing MongoDB instance

4. Run the Service

   cargo run

5. Access the API

   • API: http://localhost:3000
   • Swagger UI: http://localhost:3000/documentation
   • Health Check: http://localhost:3000/health

Using Docker

1. Build and Run

   docker build -t crud-service-rust .
   docker run -p 3000:3000 \
     -e MONGODB_URL=mongodb://host.docker.internal:27017/crud_service_rust \
     -e COLLECTION_DEFINITION_FOLDER=/app/collections \
     -e USER_ID_HEADER_KEY=userid \
     -v $(pwd)/examples/collections:/app/collections \
     crud-service-rust

2. Using Docker Compose

   # docker-compose.yml
   version: '3.8'
   services:
     mongodb:
       image: mongo:7.0
       ports:
         - "27017:27017"
       volumes:
         - mongodb_data:/data/db
     
     crud-service:
       build: .
       ports:
         - "3000:3000"
       depends_on:
         - mongodb
       environment:
         MONGODB_URL: mongodb://mongodb:27017/crud_service_rust
         COLLECTION_DEFINITION_FOLDER: /app/collections
         USER_ID_HEADER_KEY: userid
       volumes:
         - ./examples/collections:/app/collections
         - ./examples/views:/app/views
   
   volumes:
     mongodb_data:

Configuration

Environment Variables

Variable	Required	Default	Description
MONGODB_URL	✓	-	MongoDB connection string
COLLECTION_DEFINITION_FOLDER	✓	-	Path to collection definitions
USER_ID_HEADER_KEY	✓	-	Header key for user identification
VIEWS_DEFINITION_FOLDER	-	-	Path to view definitions
PORT	-	3000	Server port
CRUD_MAX_LIMIT	-	200	Maximum query limit
LOG_LEVEL	-	info	Logging level

Collection Definitions

Collections are defined in JSON or JavaScript files:

{
  "name": "users",
  "endpointBasePath": "/users",
  "defaultState": "DRAFT",
  "schema": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string"
      },
      "email": {
        "type": "string",
        "format": "email"
      },
      "age": {
        "type": "number",
        "minimum": 0
      }
    },
    "required": ["name", "email"],
    "additionalProperties": false
  },
  "indexes": [
    {
      "name": "email_unique",
      "type": "normal",
      "unique": true,
      "fields": [
        {
          "name": "email",
          "order": 1
        }
      ]
    }
  ]
}

View Definitions

Views support MongoDB aggregation pipelines:

{
  "name": "active_users",
  "source": "users",
  "type": "view",
  "pipeline": [
    {
      "$match": {
        "__STATE__": "PUBLIC",
        "status": "active"
      }
    },
    {
      "$project": {
        "name": 1,
        "email": 1,
        "lastLogin": 1
      }
    }
  ]
}

API Reference

Collections

Method	Endpoint	Description
GET	/{collection}	List documents
POST	/{collection}	Create document
GET	/{collection}/{id}	Get document by ID
PATCH	/{collection}/{id}	Update document
DELETE	/{collection}/{id}	Delete document
GET	/{collection}/count	Count documents
POST	/{collection}/bulk	Bulk create
PATCH	/{collection}/bulk	Bulk update
DELETE	/{collection}/bulk	Bulk delete
POST	/{collection}/state	Change state (bulk)
POST	/{collection}/{id}/state	Change document state
GET	/{collection}/export	Export data
POST	/{collection}/import	Import data

Views

Method	Endpoint	Description
GET	/views/{view}	Get view data
GET	/views/{view}/count	Count view data
GET	/views/{view}/export	Export view data

Utility Endpoints

Method	Endpoint	Description
GET	/health	Health check
GET	/ready	Readiness check
GET	/-/schemas	Get all schemas
GET	/-/schemas/{collection}	Get collection schema
GET	/-/metrics	Prometheus metrics
GET	/documentation	Swagger UI

Advanced Features

Authentication

Support for JWT tokens and custom authentication:

# Using JWT
curl -H "Authorization: Bearer <jwt-token>" \
     http://localhost:3000/users

# Using custom header
curl -H "userid: user123" \
     http://localhost:3000/users

Access Control Lists (ACL)

Row-level and column-level access control:

# Row-level filtering
curl -H 'acl_rows: {"userId": "123"}' \
     http://localhost:3000/orders

# Column restrictions
curl -H 'acl_read_columns: ["name", "email"]' \
     http://localhost:3000/users

Query Parameters

Rich querying capabilities:

# Filtering
GET /users?filter={"status":"active"}&limit=10&skip=20

# Sorting
GET /users?sort={"createdAt":-1}

# Projection
GET /users?projection={"name":1,"email":1}

# State filtering
GET /users?state=PUBLIC

Import/Export

Multiple format support:

# Export as JSON
GET /users/export?format=json

# Export as CSV
GET /users/export?format=csv

# Import from file
POST /users/import
Content-Type: multipart/form-data

Performance

Benchmarks

The Rust implementation provides significant performance improvements:

• Throughput: 2-3x higher requests per second
• Latency: 40-60% lower response times
• Memory: 50-70% lower memory usage
• CPU: More efficient CPU utilization

Optimization Features

• Connection pooling
• Query optimization
• Efficient serialization
• Streaming responses
• Compression support

Development

Building

# Debug build
cargo build

# Release build
cargo build --release

# Run tests
cargo test

# Format code
cargo fmt

# Lint
cargo clippy

Testing

# Unit tests
cargo test

# Integration tests (requires MongoDB)
cargo test --test integration

# With coverage
cargo tarpaulin --out html

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Run the test suite
6. Submit a pull request

Deployment

Production Deployment

# Build optimized binary
cargo build --release

# Run with production settings
RUST_LOG=info ./target/release/crud-service-rust

Docker Deployment

# Build image
docker build -t crud-service-rust .

# Run container
docker run -d \
  --name crud-service \
  -p 3000:3000 \
  -e MONGODB_URL=mongodb://mongodb:27017/production \
  crud-service-rust

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crud-service-rust
spec:
  replicas: 3
  selector:
    matchLabels:
      app: crud-service-rust
  template:
    metadata:
      labels:
        app: crud-service-rust
    spec:
      containers:
      - name: crud-service-rust
        image: crud-service-rust:latest
        ports:
        - containerPort: 3000
        env:
        - name: MONGODB_URL
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: url
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
        readinessProbe:
          httpGet:
            path: /ready
            port: 3000

Monitoring

Health Checks

• /health: Service health status
• /ready: Service readiness status

Metrics

Prometheus metrics available at /-/metrics:

• Request counts and latencies
• Database connection status
• Memory and CPU usage
• Custom business metrics

Logging

Structured JSON logging with configurable levels:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "INFO",
  "message": "Document created",
  "collection": "users",
  "document_id": "60f7b3b4e1b4c8d9a0123456",
  "user_id": "user123",
  "request_id": "req_abc123"
}

License

Apache 2.0 License - see LICENSE file for details.

Support

• Documentation: docs/
• Issues: GitHub Issues
• Discussions: GitHub Discussions