add(capability): Data Processing & ETL (#957)

Author: smendis-scottlogic

architectures/rag-llm-app/README.md

@@ -71,7 +71,7 @@ This document maps the services used in a Retrieval-Augmented Generation (RAG) L
 
 | **Component**              | **Google Cloud**          | **Amazon Web Services (AWS)** | **Microsoft Azure**             | **CCC Service**                                  |
 | -------------------------- | ------------------------- | ----------------------------- | ------------------------------- | ------------------------------------------------ |
-| **ETL/Data Processing**    | Dataflow                  | Glue, Lambda                  | Azure Data Factory              | Data Processing (Service not yet defined)        |
+| **ETL/Data Processing**    | Dataflow                  | Glue                          | Azure Data Factory              | [CCC.ETL](/catalogs/orchestration/etl/)        |
 | **Workflow Orchestration** | Cloud Composer (Airflow)  | Step Functions                | Durable Functions, Logic Apps   | Workflow Orchestration (Service not yet defined) |
 | **Chunking & Indexing**    | Cloud Composer w/ Airflow | Glue jobs                     | Data Factory Mapping Data Flows | Data Processing (Service not yet defined)        |
 

catalogs/orchestration/etl/capabilities.yaml

@@ -0,0 +1,116 @@
+imported-capabilities:
+  - reference-id: CCC
+    entries:
+      - reference-id: CCC.Core.CP01
+        remarks: Encryption in Transit
+      - reference-id: CCC.Core.CP02
+        remarks: Encryption at Rest
+      - reference-id: CCC.Core.CP03
+        remarks: Access Log Publication
+      - reference-id: CCC.Core.CP06
+        remarks: Access control
+      - reference-id: CCC.Core.CP07
+        remarks: Event Publication
+      - reference-id: CCC.Core.CP09
+        remarks: Metrics Publication
+      - reference-id: CCC.Core.CP10
+        remarks: Log Publication
+      - reference-id: CCC.Core.CP14
+        remarks: API Access
+      - reference-id: CCC.Core.CP18
+        remarks: Resource Versioning
+      - reference-id: CCC.Core.CP20
+        remarks: Resource Tagging
+      - reference-id: CCC.Core.CP23
+        remarks: Network Access Rules
+      - reference-id: CCC.Core.CP28
+        remarks: Command-line Interface
+      - reference-id: CCC.Core.CP29
+        remarks: Active Ingestion
+      - reference-id: CCC.Core.CP31
+        remarks: Elastic Scaling
+
+capabilities:
+  - id: CCC.ETL.CP01
+    title: Batch Processing
+    description: |
+      Supports the processing of bounded (batch) data sources
+      using a consistent programming model or engine.
+  
+  - id: CCC.ETL.CP02
+    title: Stream Processing
+    description: |
+      Supports the processing of unbounded (streaming) data sources
+      using a consistent programming model or engine.
+
+  - id: CCC.ETL.CP03
+    title: Schema Evolution
+    description: |
+      Automatically detects source data structures and manages changes in
+      schema (e.g., column additions) over time without pipeline failure.
+
+  - id: CCC.ETL.CP04
+    title: Distributed Data Shuffling
+    description: |
+      Provides an internal service to re-partition and group data across
+      distributed workers for complex operations like joins and aggregations.
+
+  - id: CCC.ETL.CP05
+    title: Windowing and Event-Time Processing
+    description: |
+      Enables grouping of data based on time attributes, supporting tumbling,
+      hopping, and session windows with late-data handling (watermarking).
+
+  - id: CCC.ETL.CP06
+    title: Change Data Capture (CDC) Integration
+    description: |
+      Supports incremental data ingestion by tracking changes in source
+      transaction logs rather than full table scans.
+
+  - id: CCC.ETL.CP07
+    title: Connectivity and Connector Library
+    description: |
+      Provides pre-built, managed connectors for a variety of sources and
+      sinks (e.g., Object Storage, RDBMS, NoSQL, Pub/Sub).
+
+  - id: CCC.ETL.CP08
+    title: Job Bookmarks
+    description: |
+      Persists the state of a processing job (e.g., checkpointing or bookmarks)
+      to ensure exactly-once processing and fault tolerance.
+
+  - id: CCC.ETL.CP09
+    title: Pushdown Optimization
+    description: |
+      The ability to translate transformation logic into the native language
+      of the source or sink (e.g., SQL) to minimize data movement.
+
+  - id: CCC.ETL.CP10
+    title: Visual Orchestration
+    description: |
+      Provides a graphical interface to define dependencies
+      between extraction, transformation, and loading tasks.
+
+  - id: CCC.ETL.CP11
+    title: Data Lineage & Metadata Tracking
+    description: |
+      Captures and exports metadata regarding the data sources,
+      the transformation steps, and the final destination (sink), showing the
+      flow from source to destination for compliance and debugging.
+
+  - id: CCC.ETL.CP12
+    title: User-Defined Function (UDF) Support
+    description: |
+      Allows developers to inject custom logic (Python, Java, SQL) into the
+      managed processing pipeline for complex transformations.
+
+  - id: CCC.ETL.CP13
+    title: Time-Based Job Triggering
+    description: |
+      Supports time-based (cron) mechanisms to initiate data processing workflows.
+
+  - id: CCC.ETL.CP14
+    title: Event Based Job Triggering
+    description: |
+      Supports event-based (file arrival) mechanisms to initiate
+      data processing workflows.

catalogs/orchestration/etl/metadata.yaml

@@ -0,0 +1,24 @@
+metadata:
+  id: CCC.ETL
+  title: CCC ETL
+  description: |
+    Service provides capabilities for extracting, transforming, and loading (ETL) data 
+    across diverse sources and sinks. It supports batch and real-time streaming 
+    architectures, managed data orchestration (DAGs), and serverless execution 
+    engines to process large-scale datasets with built-in fault tolerance.
+  category-ids: 
+    - CCC.Pipeline
+  version: ""
+  last-modified: "2026-02-12T00:00:00Z"
+  example-csp-services:
+    - provider: AWS
+      service: AWS Glue
+      url: https://docs.aws.amazon.com/glue/
+    - provider: Azure
+      service: Azure Data Factory
+      url: https://learn.microsoft.com/azure/data-factory/
+    - provider: GCP
+      service: Google Cloud Dataflow
+      url: https://cloud.google.com/dataflow/docs
+  applicability-categories: []
+  mapping-references: []