Context Diagram

graph TD
  subgraph Distributors
  A((API)) -->|provide data| D
  B((File Storage)) -->|drop files| D
  C((Email)) -->|send data| D
  D((Distributor)) --> E((Data Integration Platform))
  end

  subgraph Data Integration Platform
  E((Data Integration Platform)) --> F((Data Store))
  F --> E
  E --> G((User Interface))
  end

  subgraph Users
  G -->|access UI| H((Admins))
  G -->|access UI| I((Operators))
  end

  subgraph Alert Channels
  J((Email)) -.-> D
  K((SMS)) -.-> D
  L((Microsoft Teams)) -.-> D
  M((Slack)) -.-> D
  end

Loading

Container Diagram

graph TD
  subgraph Distributors
  A((API)) -->|provide data| D
  B((File Storage)) -->|drop files| D
  C((Email)) -->|send data| D
  D((Distributor)) --> E((Data Integration Platform))
  end

  subgraph Data Integration Platform
  E((Data Integration Platform)) --> F((Data Store))
  F --> E
  E --> G((API Gateway))
  G --> H((Data Collection Service))
  H --> I((Data Validation Service))
  I --> J((Data Transformation Service))
  J --> K((Data Storage Service))
  E --> L((User Interface))
  L --> M((User Management Service))
  L --> N((Logging Service))
  L --> O((Alert Service))
  end

  subgraph Users
  L --> P((Admin UI))
  L --> Q((Operator UI))
  end

  subgraph Alert Channels
  R((Email)) -.-> O
  S((SMS)) -.-> O
  T((Microsoft Teams)) -.-> O
  U((Slack)) -.-> O
  end

Loading

Containers

• Data Collection Service: This service is responsible for collecting data from different sources, such as APIs, file storage systems, and email. It communicates with each distributor to get the data documents and validate them for format, content, and timing.

• Data Validation Service: This service is responsible for validating the data documents according to defined business rules. It communicates with the Data Collection Service to get the documents and returns alerts to the Alert Service in case of issues.

• Data Transformation Service: This service is responsible for transforming the data documents into a common format for analysis. It communicates with the Data Validation Service to get the validated documents and transforms them into the target format.

• Data Storage Service: This service is responsible for storing the transformed data documents in the data store (data lake). It communicates with the Data Transformation Service to get the transformed documents and stores them in the appropriate location.

• API Gateway: This component is responsible for handling the incoming requests from the distributors and routing them to the appropriate service.

• User Interface: This component provides a UI for configuring the file transfers and defining the validation, timing, and business rules.

• User Management Service: This service is responsible for managing users and roles within the application.

• Logging Service: This service

More

  sequenceDiagram
    participant UI
    participant Distributors
    participant Data Store
    participant Admins

    UI->>Distributors: Retrieve Documents
    Distributors->>UI: Deliver Documents
    UI->>Data Store: Validate Documents
    Data Store->>Data Store: Transform Data
    Data Store->>Data Store: Store Data
    UI->>Admins: User Management

    Note over UI,Data Store: Set up File Transfer
    Note over UI,Data Store: Define Document Format
    Note over UI,Data Store: Define Consumption Schedule
    Note over UI,Data Store: Map Source Document to Data Store
    Note over UI,Data Store: Define Validation, Timing and Business Rules
    Note over Data Store: Display Logs and Metrics

Loading

System Components

• User Interface The User Interface is responsible for the configuration of the File Transfer, Document Format, Consumption Schedule, Field Mapping, and Rules for Validation, Timing, and Business Rules. The UI also provides the Dashboard to view the logs and metrics of the Data Integration Platform.

• Distributors The Distributors are responsible for providing Data Documents to the Platform. The Data Documents can be in various formats and delivered through different methods, such as API, File Storage Systems, Email, Document Management Systems, or the internal File Storage System provided by the Platform.

• Data Store The Data Store is responsible for storing Data Documents and performing transformations on the data to move it to a common format for analysis and possible further transformation. It also validates the Documents for Format, Content, Timing, and other Business Rules defined by the User Interface.

• Admins The Admins are responsible for User Management in the Data Integration Platform. They can create Users and Roles that can perform and view different components and actions within the Platform.

• Conclusion The Data Integration Platform is designed to handle disparate sets of data documents from different sources and deliver them to a common format for analysis. The Platform is flexible and can consume data from various sources, validate the data, transform it to a common format, and store it in a Data Store for analysis. The User Interface provides the configuration and monitoring of the Platform, while the Admins provide the User Management.