Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
1246 lines (985 loc) · 41.4 KB

ARCHITECTURE.md

File metadata and controls

1246 lines (985 loc) · 41.4 KB
Raw

Architecture Document: YouTube Success Prediction ML Platform

Python pandas NumPy scikit-learn SciPy joblib Pydantic FastAPI Uvicorn Flask Plotly Folium python-dotenv pytest HTTPX Ruff Black + iSort Node.js npm Next.js React TypeScript Recharts ESLint Prettier Docker Docker Compose Kubernetes Kustomize Argo CD Argo Rollouts Terraform Jenkins GitHub Actions AWS Google Cloud Azure Oracle Cloud GNU Make Bash Mermaid Vercel MLflow Weights & Biases Optuna DVC Feast Prefect Prometheus Grafana

Table Of Contents

Document Metadata

Field Value
Document role System architecture and engineering design authority
Primary audience Senior ML/backend/frontend/platform engineers and technical reviewers
Last updated March 8, 2026
Operational companion README.md
Contract companion API_REFERENCE.md

Documentation Map

Document Scope Use it when
README.md Operational bootstrap and runbook You need setup, quality gates, and execution commands
API_REFERENCE.md Endpoint contracts and constraints You need payload semantics and error behavior
MLOPS.md Lineage and governance operations You need manifest/registry/drift details
DEPLOYMENT.md Delivery and rollout strategy You need CI/CD and release orchestration guidance
FRONTEND.md Client routes and API consumption model You need frontend integration boundaries
infra/README.md Infrastructure topology index You need infra-level navigation and quick commands
infra/k8s/README.md Kubernetes runtime objects You are changing manifests, policies, or overlays
infra/argocd/README.md GitOps application control You are switching sync strategy or app definitions
infra/terraform/README.md Cloud provisioning model You are planning/applying infrastructure packs

1. Purpose

This document describes the technical architecture of the YouTube Success Prediction ML Platform. It covers:

  • system boundaries
  • runtime components
  • data and model lifecycles
  • API and frontend integration
  • MLOps lineage and observability
  • operational reliability patterns

The architecture is designed to be portfolio-grade while staying executable in a local environment.

2. Scope

In Scope

  • Supervised inference for subscriber/earnings/growth predictions.
  • Unsupervised clustering and archetype labeling.
  • Country-level metrics and visualization-ready data delivery.
  • FastAPI and Flask runtime services.
  • Next.js frontend for interactive analytics.
  • Training-time MLOps artifacts and registry.
  • Optional experiment tracking (MLflow, W&B) and HPO (Optuna) integration.
  • Feature-store and data-versioning scaffolding (Feast, DVC).
  • Scheduled retraining orchestration (Prefect) and deployable monitoring stack (Prometheus, Grafana).

Out Of Scope

  • multi-tenant identity and RBAC.
  • distributed retraining orchestration.
  • managed online feature store serving tier.
  • managed model serving platform.

3. Architectural Goals

  • Reproducibility: deterministic training metadata and artifact hashes.
  • Operability: health/readiness endpoints plus metrics exposure.
  • Modularity: clear separation between data, modeling, API, and UI layers.
  • Replaceability: model and serving layers can evolve independently.
  • Traceability: each training run tied to data fingerprint and config snapshot.

4. High-Level System Context

flowchart TB
    User[Product User] --> FE[Next.js Frontend]
    FE --> API[FastAPI Service]
    FE --> API2[Flask Service]

    API --> Models[(Model Artifacts)]
    API2 --> Models

    Train[Training CLI] --> Data[(Raw CSV Dataset)]
    Train --> Models
    Train --> Reports[(Metrics and Data Quality Reports)]
    Train --> Registry[(Manifest and Model Registry)]

    API --> Registry
    API2 --> Registry
Loading

5. Container View

flowchart LR
    subgraph Browser
        UI[Next.js App Routes]
    end

    subgraph Backend
        FAPI[FastAPI App]
        FLASK[Flask App]
        SVC[Predictor and Cluster Services]
        MLOPS[MLOps Registry Utilities]
    end

    subgraph ML
        TRAIN[train.py]
        SUP[Supervised Pipeline]
        CLU[Clustering Pipeline]
        MAP[Map Builders]
    end

    subgraph Storage
        CSV[(Global YouTube CSV)]
        ART[(artifacts/models)]
        REP[(artifacts/reports)]
        REG[(artifacts/mlops)]
    end

    UI --> FAPI
    UI --> FLASK
    FAPI --> SVC
    FLASK --> SVC
    SVC --> ART
    FAPI --> MLOPS

    TRAIN --> CSV
    TRAIN --> SUP
    TRAIN --> CLU
    TRAIN --> MAP
    SUP --> ART
    CLU --> ART
    MAP --> REP
    TRAIN --> REP
    TRAIN --> REG
Loading

6. Component View: Backend

flowchart TD
    App[FastAPI or Flask App] --> Router[Route Layer]
    Router --> Validate[Pydantic Request Validation]
    Router --> PredictorService
    Router --> ClusterService
    Router --> Analytics[Country and Sample Data Endpoints]
    Router --> MLOpsRoutes[Manifest Registry Metrics Readiness]

    PredictorService --> SupervisedBundle[(supervised_bundle.joblib)]
    ClusterService --> ClusteringBundle[(clustering_bundle.joblib)]

    Analytics --> Loader[Data Loader]
    Loader --> CSV[(Raw Dataset)]

    MLOpsRoutes --> Manifest[(training_manifest.json)]
    MLOpsRoutes --> Registry[(model_registry.json)]
Loading

7. Data Flow: Training Lifecycle

sequenceDiagram
    participant CLI as train.py
    participant Loader as data.loader
    participant Sup as models.supervised
    participant Clu as models.clustering
    participant Maps as visualization.maps
    participant Quality as mlops.quality
    participant Registry as mlops.registry
    participant Artifacts as artifacts/*

    CLI->>Loader: load_raw_dataset()
    CLI->>Loader: load_dataset()

    CLI->>Sup: train_supervised_bundle(df, config)
    Sup-->>Artifacts: supervised_bundle.joblib

    CLI->>Clu: train_clustering_bundle(df, config)
    Clu-->>Artifacts: clustering_bundle.joblib + clustered_channels.csv

    CLI->>Maps: export_map_assets(df)
    Maps-->>Artifacts: map html outputs

    CLI->>Quality: build_data_quality_report(raw_df, processed_df)
    Quality-->>Artifacts: data_quality_report.json

    CLI->>Registry: build_training_manifest(...)
    Registry-->>Artifacts: training_manifest.json
    CLI->>Registry: update_registry(manifest)
    Registry-->>Artifacts: model_registry.json
Loading

8. Data Flow: Inference Request Lifecycle

sequenceDiagram
    participant User
    participant FE as Next.js
    participant API as FastAPI
    participant Schema as Pydantic Schema
    participant Svc as PredictorService
    participant Model as SupervisedBundle

    User->>FE: Submit uploads/category/country/age
    FE->>API: POST /predict
    API->>Schema: Validate payload
    API->>Svc: predict(request)
    Svc->>Model: models[target].predict(features)
    Model-->>Svc: subscribers/earnings/growth
    Svc-->>API: formatted response
    API-->>FE: JSON
    FE-->>User: Render prediction cards
Loading

9. Data Layer And Feature Engineering

9.1 Dataset Overview

Primary source dataset:

  • path resolution: resolve_data_path() in src/youtube_success_ml/data/loader.py
  • default source file: data/Global YouTube Statistics.csv
  • canonical source: Kaggle - Global YouTube Statistics 2023
  • optional override: YTS_DATA_PATH
  • encoding: latin-1
  • row count: 995
  • raw column count: 28

Processed training frame:

  • row count: 995
  • processed column count: 30
  • engineered columns include age and growth_target

Dataset domains:

Domain Examples
Channel identity and taxonomy youtuber, title, channel_type, category, country, abbreviation
Core performance uploads, subscribers, video_views, video_views_for_the_last_30_days
Monetization lowest_monthly_earnings, highest_monthly_earnings, lowest_yearly_earnings, highest_yearly_earnings
Growth and lifecycle subscribers_for_last_30_days, created_year, created_month, created_date, engineered age, engineered growth_target
Geo and socio-economic context latitude, longitude, population, urban_population, unemployment_rate, gross_tertiary_education_enrollment_pct

Feature/target contract for supervised inference:

  • features: uploads, category, country, age
  • targets: subscribers, highest_yearly_earnings, growth_target

9.2 Canonical Transformation Rules

Defined in src/youtube_success_ml/data/loader.py:

  • normalize column names to snake_case.
  • coerce known numeric columns with errors='coerce'.
  • fill country, category, abbreviation nulls.
  • engineer age = current_year - created_year clipped to non-negative.
  • derive growth_target = subscribers_for_last_30_days.
  • enforce non-negative clipping for key numeric targets.

9.3 Raw vs Processed Contracts

  • raw contract endpoint: /data/raw-sample
  • processed contract endpoint: /data/processed-sample

This split makes preprocessing explicit to product and stakeholders.

10. Modeling Architecture

Supervised Stack

Implementation: src/youtube_success_ml/models/supervised.py

  • Feature pipeline:
    • numeric imputation for uploads and age
    • categorical imputation and one-hot encoding for category and country
  • Regressor:
    • RandomForestRegressor
  • Target transform:
    • TransformedTargetRegressor with log1p/expm1
  • Targets:
    • subscribers
    • yearly earnings
    • 30-day growth
  • Evaluation metrics:
    • MAE
    • RMSE
    • R2

Clustering Stack

Implementation: src/youtube_success_ml/models/clustering.py

  • KMeans pipeline:
    • StandardScaler + KMeans
  • DBSCAN pipeline:
    • StandardScaler + DBSCAN
  • Profile generation:
    • aggregate per cluster (avg_uploads, avg_subscribers, avg_earnings, avg_growth)
  • Archetype naming logic:
    • highest growth => Viral entertainers
    • highest earnings per upload => High earning low upload
    • high uploads relative to growth => High upload low growth
    • remaining => Consistent educators

11. API Design

FastAPI Endpoints

  • GET /health
  • GET /ready
  • POST /predict
  • POST /predict/batch
  • POST /predict/simulate
  • POST /predict/recommendation
  • GET /predict/feature-importance
  • GET /clusters/summary
  • GET /maps/country-metrics
  • GET /maps/influence-map
  • GET /maps/earnings-choropleth
  • GET /maps/category-dominance
  • GET /data/raw-sample
  • GET /data/processed-sample
  • GET /analytics/category-performance
  • GET /analytics/upload-growth-buckets
  • GET /mlops/manifest
  • GET /mlops/registry
  • POST /mlops/drift-check
  • GET /mlops/capabilities
  • GET /metrics

Flask Endpoints

Equivalent functional prediction, analytics, and MLOps endpoints are exposed via Flask for framework portability. Prometheus-style metrics are implemented in FastAPI.

Readiness Strategy

/ready checks artifact existence through check_artifacts_ready().

If critical artifacts are missing, service reports not-ready state, allowing load balancers or orchestrators to block traffic.

12. Frontend Architecture

Routes

  • / main dashboard with prediction workflow, archetype table, and six visual insight cards above Global Country Intelligence
  • /visualizations/charts real world map embeds + analytics + raw/processed data comparisons
  • /intelligence/lab simulation, recommendation, explainability, four insight cards above Batch Prediction Workbench, and drift analysis (with run-lab empty-state chart guidance)
  • /wiki embedded project wiki route
  • /wiki/index.html standalone static wiki landing page

Client Data Access

frontend/lib/api.ts centralizes typed API requests and transport error handling.

UI Composition

  • composable chart + table sections
  • typed data contracts in frontend/lib/types.ts
  • progressive loading and error display for asynchronous calls
  • backend-driven iframe map embeds for high-fidelity world map rendering
  • overview visuals include market momentum, archetype-share, revenue-efficiency, category-pressure, market-share-balance, and monetization-lift cards above country intelligence
  • model lab includes growth-elasticity, explainability-concentration, earnings-response, and drift-severity cards above batch workbench
  • drift snapshot panel remains visible at all times; explicit run-lab guidance renders when idle and skeletons are used only during active lab execution
  • icon-only top-left shell control toggles animated collapse/expand of the sticky top navbar
  • dedicated responsive navigation and layout shell for desktop/mobile parity

Frontend Visual Data Dependencies

flowchart LR
    CountryMetrics["/maps/country-metrics"] --> OverviewCards["Overview intelligence cards"]
    ClusterSummary["/clusters/summary"] --> OverviewCards
    Simulate["/predict/simulate"] --> LabCards["Growth and earnings lab cards"]
    FeatureImportance["/predict/feature-importance"] --> LabCards
    DriftCheck["/mlops/drift-check"] --> DriftPanels["Drift snapshot + severity mix"]
Loading

Frontend Shell Interaction States

stateDiagram-v2
    [*] --> NavExpanded
    NavExpanded --> NavCollapsed: top-left icon toggle
    NavCollapsed --> NavExpanded: top-left icon toggle
    NavExpanded --> NavMenuOpen: mobile Menu toggle
    NavMenuOpen --> NavExpanded: close or route change
Loading

SEO And Metadata Integration

  • global metadata and social previews in frontend/app/layout.tsx
  • robots and sitemap generation in frontend/app/robots.ts and frontend/app/sitemap.ts
  • web manifest in frontend/app/manifest.ts
  • favicon and icon asset pipeline via frontend/public/

13. MLOps Architecture

Artifact Lineage

flowchart LR
    D[Dataset SHA256] --> M[training_manifest.json]
    C[Training Config Snapshot] --> M
    R[Model and Report Artifact Hashes] --> M
    FS[feature_store_snapshot.csv] --> M
    M --> G[model_registry.json]
    G --> A[active_run_id]
Loading

Outputs

  • metrics report
  • data quality report
  • feature snapshot report
  • training manifest
  • model registry with active run pointer

Extended MLOps Modules

flowchart TB
    TRAIN[run_training] --> EXP[Experiment Tracker]
    TRAIN --> HPO[Optuna HPO]
    TRAIN --> QUAL[Quality + Drift Baseline]
    TRAIN --> SNAP[Feature Snapshot Export]
    QUAL --> REG[Manifest + Registry]
    SNAP --> FEAST[Feast Repo Assets]
    SNAP --> DVC[DVC Pipeline Outputs]
    EXP --> LOGS[MLflow / W&B]
Loading

Extension activation is opt-in so the default training/testing path remains stable without additional services.

Operational Benefits

  • reproducible run tracing
  • easier rollback decisions
  • audit-friendly history of model state transitions
  • controlled optionality for advanced MLOps tooling

14. Observability

Implemented

  • health endpoint
  • readiness endpoint
  • path-level request counters and cumulative latency (/metrics)
  • process-time response header (X-Process-Time-Seconds)
  • in-repo Prometheus configuration (infra/monitoring/prometheus)
  • in-repo Grafana provisioning and dashboards (infra/monitoring/grafana)
  • Kubernetes monitoring overlay (infra/k8s/monitoring)

Runtime Capability Surface

  • GET /mlops/capabilities provides runtime posture:
    • installed optional packages (mlflow, wandb, optuna, prefect)
    • presence of feature-store/data-versioning files
    • presence of monitoring assets

15. Reliability And Failure Modes

Failure: Missing Artifacts

Symptom:

  • /predict returns service unavailable.

Mitigation:

  • run training pipeline.
  • verify /ready returns HTTP 200.

Failure: Input Schema Drift

Symptom:

  • validation errors on /predict.

Mitigation:

  • strict Pydantic contract.
  • frontend type alignment in frontend/lib/types.ts.

Failure: Dataset Parsing Issues

Symptom:

  • load or coercion errors.

Mitigation:

  • explicit encoding and numeric coercion strategy in loader.

16. Security Posture

Implemented

  • schema-level input validation.
  • bounded sample limits on raw/processed data endpoints.

Recommended Hardening

  • auth and token validation.
  • rate limiting and request throttling.
  • network policy and TLS termination.
  • secret management with environment or vault integration.

17. Deployment Topology

flowchart TB
    subgraph Runtime
        LB[Ingress or Reverse Proxy]
        FE[Next.js Container]
        API[FastAPI Container]
    end

    subgraph Storage
        Artifacts[(Persistent Volume: artifacts)]
        Data[(Dataset Volume)]
    end

    LB --> FE
    LB --> API
    API --> Artifacts
    API --> Data
Loading

Local Container Stack

  • docker/Dockerfile.api
  • docker/Dockerfile.frontend
  • docker-compose.yml

18. CI/CD Model

This repository supports two complementary CI/CD planes:

  • Pull request validation with GitHub Actions (.github/workflows/ci.yml)
  • Release delivery with Jenkins + Argo CD + Argo Rollouts (Jenkinsfile, infra/argocd, infra/k8s/overlays)

Pipeline responsibilities:

  • GitHub Actions:
    • Python dependency install, training pipeline execution, and pytest
    • frontend install, lint, and production build
    • intentionally uses baseline training mode (no required MLflow/Optuna/Prefect services)
  • Jenkins:
    • repeat quality gates (train/test/lint/build)
    • container build/push to cloud registry
    • overlay image updates for selected rollout strategy
    • terraform plan/apply orchestration per cloud provider
    • Argo CD sync and optional blue/green promotion gate

Extension strategy:

  • Advanced MLOps modules are additive and activated by explicit operator intent.
  • This prevents CI fragility while preserving production extensibility.

This split keeps PR quality checks fast while preserving production-grade release controls.

GitHub Actions Workflow

19. Capacity And Scalability Notes

Current Runtime Profile

  • batch training on local CSV.
  • in-memory inference with preloaded artifacts.
  • stateless API process except filesystem artifact reads.

Horizontal Scaling

  • API can scale horizontally if artifact directory is shared or baked into immutable image.
  • frontend is static-app friendly and cacheable.

Vertical Scaling

  • train-time forest size and clustering config tunable via env vars.
  • CPU-bound inference can be optimized with model simplification or serving workers.

20. Decisions And Tradeoffs

  • RandomForest chosen for robust nonlinear performance on mixed feature types and small/medium dataset size.
  • separate target models for clarity and independent metric tracking.
  • filesystem-based registry chosen for simplicity and portability in local/portfolio contexts.
  • dual API frameworks included to demonstrate framework portability and service abstraction.

21. Suggested Enterprise Evolution Path

  • replace local registry files with managed model registry.
  • evolve Feast scaffolding into managed offline/online feature serving.
  • harden Prefect retraining schedules with approval workflows and policy checks.
  • add canary release process for promoted models.
  • extend drift checks from request-level to population-level production windows.
  • add SLO dashboards and alerting tied to /metrics and model KPI regressions.

22. Source File Index

Core files:

  • src/youtube_success_ml/train.py
  • src/youtube_success_ml/data/loader.py
  • src/youtube_success_ml/models/supervised.py
  • src/youtube_success_ml/models/clustering.py
  • src/youtube_success_ml/api/fastapi_app.py
  • src/youtube_success_ml/api/flask_app.py
  • src/youtube_success_ml/mlops/quality.py
  • src/youtube_success_ml/mlops/registry.py
  • src/youtube_success_ml/mlops/experiments.py
  • src/youtube_success_ml/mlops/hpo.py
  • src/youtube_success_ml/mlops/feature_store.py
  • frontend/app/page.tsx
  • frontend/app/visualizations/charts/page.tsx
  • frontend/app/intelligence/lab/page.tsx
  • frontend/app/wiki/page.tsx
  • frontend/public/wiki/index.html
  • .devcontainer/devcontainer.json
  • .devcontainer/post-create.sh
  • scripts/format_all.sh
  • scripts/format_prettier.sh
  • scripts/format_python.sh
  • scripts/mlops/export_feature_store_snapshot.py
  • scripts/mlops/run_prefect_retraining.sh
  • orchestration/prefect/retraining_flow.py
  • dvc.yaml
  • params.yaml
  • feature_store/feast/feature_store.yaml
  • feature_store/feast/feature_repo.py
  • infra/monitoring/prometheus/prometheus.yml
  • infra/monitoring/grafana/dashboards/yts-api-observability.json
  • infra/k8s/monitoring/kustomization.yaml

23. Operational Checklist

Before release:

  • run training successfully
  • verify /ready is healthy
  • run full pytest suite
  • run frontend lint and build
  • confirm manifest and registry updated
  • confirm smoke test passes against deployed API

24. Extended Interaction Charts

API Capability Topology

flowchart LR
    Client[Client Apps] --> H["/health"]
    Client --> R["/ready"]

    Client --> P["/predict"]
    Client --> PB["/predict/batch"]
    Client --> PS["/predict/simulate"]
    Client --> PR["/predict/recommendation"]
    Client --> FI["/predict/feature-importance"]

    Client --> CS["/clusters/summary"]
    Client --> CM["/maps/country-metrics"]
    Client --> MI["/maps/influence-map"]
    Client --> ME["/maps/earnings-choropleth"]
    Client --> MD["/maps/category-dominance"]
    Client --> RS["/data/raw-sample"]
    Client --> PSM["/data/processed-sample"]

    Client --> MF["/mlops/manifest"]
    Client --> MR["/mlops/registry"]
    Client --> DC["/mlops/drift-check"]
    Client --> CAP["/mlops/capabilities"]
    Client --> MT["/metrics"]

    P --> SVC[IntelligenceService]
    PB --> SVC
    PS --> SVC
    PR --> SVC
    FI --> SVC
    CS --> SVC
    DC --> SVC
Loading

Internal Service Collaboration

sequenceDiagram
    participant Router as API Router
    participant IS as IntelligenceService
    participant SB as SupervisedBundle
    participant CB as ClusteringBundle
    participant BL as Drift Baseline

    Router->>IS: predict / batch / simulate / recommendation
    IS->>SB: run target regressors
    SB-->>IS: subscribers, earnings, growth

    IS->>CB: cluster projection for recommendation
    CB-->>IS: cluster_id, archetype

    Router->>IS: drift_check(items)
    IS->>BL: compare distributions
    BL-->>IS: severity records + summary
    IS-->>Router: typed response payload
Loading

Failure Handling Matrix

flowchart TD
    A[Incoming Request] --> B{Validation OK?}
    B -- No --> E[400 Validation Error]
    B -- Yes --> C{Artifacts Ready?}
    C -- No --> F[503 Service Unavailable]
    C -- Yes --> D{Endpoint Logic Success?}
    D -- No --> G[4xx/5xx with structured detail]
    D -- Yes --> H[200 Successful Response]
Loading

25. Data Contracts By Layer

Inference Input Contract

classDiagram
    class PredictionRequest {
      +int uploads
      +string category
      +string country
      +int age
    }

    class PredictionResponse {
      +float predicted_subscribers
      +float predicted_earnings
      +float predicted_growth
    }

    PredictionRequest --> PredictionResponse
Loading

Artifact Contract

classDiagram
    class TrainingManifest {
      +string run_id
      +string timestamp_utc
      +string data_sha256
      +object training_config
      +object metrics
      +object artifact_hashes
      +object artifact_paths
    }

    class ModelRegistry {
      +string active_run_id
      +list runs
    }

    TrainingManifest --> ModelRegistry : updates active run
Loading

26. Deployment Variants

flowchart LR
    subgraph LocalDev
      DEVFE[Next.js dev server]
      DEVAPI[FastAPI local process]
      DEVART[(Local artifacts)]
    end

    subgraph Containerized
      CFE[Frontend container]
      CAPI[API container]
      CART[(Mounted artifacts volume)]
    end

    DEVFE --> DEVAPI
    DEVAPI --> DEVART

    CFE --> CAPI
    CAPI --> CART
Loading

27. Incident Response Flow

sequenceDiagram
    participant Alert as Monitoring Alert
    participant SRE as Operator
    participant API as Service
    participant MLOPS as Manifest/Registry

    Alert->>SRE: Elevated error or drift risk
    SRE->>API: Check /ready and /metrics
    SRE->>API: Check /mlops/drift-check
    API-->>SRE: Drift severity and readiness
    SRE->>MLOPS: Inspect active run and artifact hashes
    SRE->>API: Redeploy or retrain decision
Loading

28. Trust Boundaries And Data Zones

flowchart LR
    subgraph PublicZone
      U[External User]
      B[Browser]
    end

    subgraph AppZone
      FE[Frontend Service]
      API[API Service]
    end

    subgraph DataZone
      RAW[(Raw Dataset)]
      ART[(Artifacts)]
      META[(Manifest/Registry)]
    end

    U --> B
    B --> FE
    FE --> API
    API --> ART
    API --> META
    API --> RAW
Loading

Boundary assumptions:

  • Public zone is untrusted.
  • App zone enforces schema validation and readiness checks.
  • Data zone contains persistent assets and should be restricted to service principals.

29. Model Promotion And Rollback Design

sequenceDiagram
    participant Train as Training Pipeline
    participant Manifest as training_manifest.json
    participant Registry as model_registry.json
    participant API as Inference API
    participant Ops as Operator

    Train->>Manifest: write run metadata + hashes
    Train->>Registry: append run, set active_run_id
    API->>Registry: read active_run_id
    API->>Manifest: verify artifact alignment
    Ops->>Registry: set prior run (rollback scenario)
    API->>Registry: reload active run on restart
Loading

30. Request Routing By Capability

flowchart TD
    IN[Incoming HTTP Request] --> R1{Path Group}
    R1 -- health --> H[health router]
    R1 -- predict --> P[predictions router]
    R1 -- analytics --> A[analytics router]
    R1 -- mlops --> M[mlops router]

    P --> S[IntelligenceService]
    A --> L[data.loader + visualization]
    M --> REG[mlops.registry + drift]
Loading

This routing split is the primary structural refactor that makes the API easier to scale and maintain.

31. Multi-Cloud Platform Topology

flowchart TB
    subgraph ControlPlane[CI/CD Control Plane]
      J[Jenkins]
      G[GitOps Repo]
      ACD[Argo CD]
    end

    subgraph CloudA[AWS]
      EKS[EKS]
      ECR[ECR]
      S3[S3 Artifacts]
    end

    subgraph CloudB[GCP]
      GKE[GKE]
      GAR[Artifact Registry]
      GCS[GCS Artifacts]
    end

    subgraph CloudC[Azure]
      AKS[AKS]
      ACR[ACR]
      BLOB[Blob Artifacts]
    end

    subgraph CloudD[OCI]
      OKE[OKE]
      OCIR[OCIR]
      OBJ[Object Storage]
    end

    J --> G
    G --> ACD

    J --> ECR
    J --> GAR
    J --> ACR
    J --> OCIR

    ACD --> EKS
    ACD --> GKE
    ACD --> AKS
    ACD --> OKE

    EKS --> S3
    GKE --> GCS
    AKS --> BLOB
    OKE --> OBJ
Loading

32. Strategy Control Plane

flowchart LR
    Select[Deployment Strategy Parameter] --> Overlay[Kustomize Overlay]
    Overlay --> ArgoApp[Argo Application]
    ArgoApp --> Runtime{Runtime Object Type}

    Runtime -- rolling --> KDeployment[Kubernetes Deployment]
    Runtime -- canary --> KRolloutCanary[Argo Rollout Canary]
    Runtime -- bluegreen --> KRolloutBG[Argo Rollout BlueGreen]
Loading

Key design:

  • One runtime namespace (yts-prod) and one baseline service contract.
  • Strategy decides controller behavior, not app code.
  • Frontend and API move together under the same release tag.

33. Terraform Composition Model

flowchart TD
    ENV[Environment Root] --> TAGS[global_tags module]
    ENV --> PLATFORM[cloud platform module]

    PLATFORM --> K8S[Managed Kubernetes]
    PLATFORM --> REG[Container Registry]
    PLATFORM --> OBJ[Artifact Storage]

    TAGS --> K8S
    TAGS --> REG
    TAGS --> OBJ
Loading

Root modules:

  • infra/terraform/environments/aws
  • infra/terraform/environments/gcp
  • infra/terraform/environments/azure
  • infra/terraform/environments/oci

Reusable modules:

  • infra/terraform/modules/global_tags
  • infra/terraform/modules/aws_platform
  • infra/terraform/modules/gcp_platform
  • infra/terraform/modules/azure_platform
  • infra/terraform/modules/oci_platform

34. Release Promotion Sequence

sequenceDiagram
    participant Dev as Developer
    participant CI as Jenkins
    participant Reg as Cloud Registry
    participant GitOps as Overlay Repo
    participant Argo as Argo CD
    participant K8s as Cluster

    Dev->>CI: push commit
    CI->>CI: train + test + frontend build
    CI->>Reg: push yts-api + yts-frontend image tags
    CI->>GitOps: update overlay image references
    CI->>Argo: sync strategy app
    Argo->>K8s: apply manifests
    K8s-->>Argo: health + rollout status
Loading

35. Rollback And Recovery Path

flowchart TD
    Alert[Alert: SLO or health regression] --> Decision{Deployment Strategy?}
    Decision -- rolling --> RollBackDeploy[kubectl rollout undo deployment]
    Decision -- canary --> AbortCanary[kubectl argo rollouts abort]
    Decision -- bluegreen --> PromoteStable[keep active service on stable ReplicaSet]

    RollBackDeploy --> Verify[Verify /health and /ready]
    AbortCanary --> Verify
    PromoteStable --> Verify
Loading

Recovery invariants:

  • health endpoints remain strategy-agnostic.
  • rollback does not require data schema changes.
  • model artifacts remain backward-compatible via manifest/registry lineage.

36. Non-Functional Requirements Matrix

Dimension Current implementation Target/SLO guidance Primary verification
Availability Health and readiness endpoints with deployment rollback paths >= 99.5% monthly API availability for single-region production /health, /ready, rollout status checks
Reliability Artifact readiness gating and versioned registry Zero silent startup with missing artifacts Readiness gate and startup smoke tests
Performance FastAPI async serving and lightweight feature preprocessing P95 prediction latency < 300ms under normal load k6/Locust load testing, Prometheus histograms
Scalability Kubernetes HPA, canary/bluegreen overlays Horizontal scaling based on CPU and request volume HPA metrics, cluster autoscaler events
Security Network policies, secret templates, no embedded cloud creds Least privilege across runtime and CI identities IaC review, container scan, secret policy checks
Observability Prometheus endpoint and drift-check diagnostics End-to-end golden signals + model-risk signal visibility /metrics, drift reports, alerting dashboards
Maintainability Layered modules with domain-separated docs Low change friction with synchronized docs and tests PR checklist and CI quality gates

37. Architecture Governance

Architecture changes follow an ADR-style workflow, even when ADR files are lightweight and embedded in pull request context.

Governance rules:

  • Any new API domain or endpoint family requires updates to API_REFERENCE.md and Section 11 of this document.
  • Any model feature/target/schema change requires updates to data contracts (Section 25), README.md, and MLOPS.md.
  • Any runtime topology change (Kubernetes, Argo, Terraform) requires updates to Section 17/18/31+ and relevant infra/* docs.
  • Breaking behavior changes must include migration notes in README.md and deployment rollback notes in DEPLOYMENT.md.
flowchart TD
    Proposal[Architecture Change Proposal] --> Impact[Impact Analysis]
    Impact --> ADR[ADR or PR Design Note]
    ADR --> Impl[Implementation]
    Impl --> Validation[Tests + Runtime Validation]
    Validation --> Docs[Documentation Synchronization]
    Docs --> Review[Architecture Review]
    Review --> Release[Release Approval]
Loading

38. Extended MLOps Control Plane

The platform now includes a layered MLOps control plane:

  • baseline lineage and drift controls (always-on)
  • optional experiment and HPO controls (MLflow/W&B/Optuna)
  • optional data lifecycle controls (DVC/Feast)
  • optional orchestration and observability controls (Prefect/Prometheus/Grafana)
flowchart LR
    subgraph Baseline
      T[Training]
      R[Manifest + Registry]
      D[Drift Check]
    end

    subgraph Optional
      E[MLflow/W&B]
      H[Optuna]
      F[DVC/Feast]
      O[Prefect]
      M[Prometheus/Grafana]
    end

    T --> R
    T --> D
    T --> E
    T --> H
    T --> F
    O --> T
    M --> D
    M --> R
Loading

39. Capability Detection Model

Capability detection is exposed via /mlops/capabilities to reduce ambiguity between environments.

Use cases:

  • deployment preflight in staging/prod
  • health dashboard enrichment
  • CI smoke checks for optional module readiness
sequenceDiagram
    participant Ops as Operator
    participant API as API Service
    participant FS as Filesystem
    participant PKG as Python Runtime

    Ops->>API: GET /mlops/capabilities
    API->>PKG: check optional imports
    API->>FS: check dvc/feast/prefect/monitoring files
    PKG-->>API: installed/not-installed map
    FS-->>API: present/missing map
    API-->>Ops: consolidated capability response
Loading

40. Monitoring And Retraining Topology

Monitoring and retraining are decoupled from request serving but integrated through common artifact and readiness contracts.

flowchart TB
    subgraph Runtime
      API[FastAPI/Flask]
      MET["/metrics"]
      READY["/ready"]
    end

    subgraph Observability
      PROM[Prometheus]
      GRAF[Grafana]
    end

    subgraph Orchestration
      PREF[Prefect Flow]
      TRAIN[run_training]
    end

    subgraph Artifacts
      REG[Manifest + Registry]
      REP[Reports + Feature Snapshot]
    end

    API --> MET
    API --> READY
    MET --> PROM
    PROM --> GRAF
    PREF --> TRAIN
    TRAIN --> REG
    TRAIN --> REP
    READY --> PREF
Loading